Oct 31 00:16:14.105528 ip-10-0-128-253 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 31 00:16:14.105540 ip-10-0-128-253 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 31 00:16:14.105546 ip-10-0-128-253 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 31 00:16:14.105782 ip-10-0-128-253 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 31 00:16:24.354808 ip-10-0-128-253 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 31 00:16:24.354824 ip-10-0-128-253 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot beefdae8801c4b93b9a97786b8dd848a --
Oct 31 00:18:17.483938 ip-10-0-128-253 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 00:18:17.923692 ip-10-0-128-253 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:18:17.923692 ip-10-0-128-253 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 00:18:17.923692 ip-10-0-128-253 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:18:17.923692 ip-10-0-128-253 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:18:17.927470 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927375    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:18:17.927470 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927466    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:18:17.927470 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927471    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:18:17.927470 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927475    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:18:17.927470 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927479    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:18:17.927801 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927486    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:18:17.927801 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927492    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:18:17.927801 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927497    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:18:17.927801 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927502    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:18:17.927801 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927506    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:18:17.927801 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927510    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:18:17.927801 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927514    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:18:17.927801 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927517    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:18:17.927801 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927521    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:18:17.927801 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927526    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:18:17.927801 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927529    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:18:17.927801 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927533    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:18:17.927801 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927542    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:18:17.927801 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927547    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:18:17.927801 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927550    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:18:17.927801 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927555    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:18:17.927801 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927558    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:18:17.927801 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927562    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:18:17.927801 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927566    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:18:17.928558 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927570    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:18:17.928558 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927574    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:18:17.928558 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927578    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:18:17.928558 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927582    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:18:17.928558 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927586    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:18:17.928558 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927589    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:18:17.928558 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927593    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:18:17.928558 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927597    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:18:17.928558 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927602    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:18:17.928558 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927606    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:18:17.928558 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927610    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:18:17.928558 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927630    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:18:17.928558 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927635    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:18:17.928558 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927639    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:18:17.928558 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927645    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:18:17.928558 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927650    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:18:17.928558 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927655    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:18:17.928558 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927659    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:18:17.928558 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927663    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:18:17.928558 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927667    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:18:17.929385 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927671    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:18:17.929385 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927676    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:18:17.929385 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927680    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:18:17.929385 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927686    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:18:17.929385 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927690    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:18:17.929385 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927695    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:18:17.929385 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927699    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:18:17.929385 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927704    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:18:17.929385 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927708    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:18:17.929385 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927712    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:18:17.929385 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927717    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:18:17.929385 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927721    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:18:17.929385 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927726    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:18:17.929385 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927730    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:18:17.929385 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927734    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:18:17.929385 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927739    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:18:17.929385 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927745    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:18:17.929385 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927752    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:18:17.929385 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927757    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:18:17.929385 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927761    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:18:17.930243 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927766    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:18:17.930243 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927773    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:18:17.930243 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927778    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:18:17.930243 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927782    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:18:17.930243 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927787    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:18:17.930243 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927792    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:18:17.930243 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927796    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:18:17.930243 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927800    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:18:17.930243 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927804    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:18:17.930243 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927809    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:18:17.930243 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927813    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:18:17.930243 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927817    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:18:17.930243 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927821    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:18:17.930243 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927826    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:18:17.930243 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927830    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:18:17.930243 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927835    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:18:17.930243 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927841    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:18:17.930243 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927845    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:18:17.930243 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927850    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:18:17.930243 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927854    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:18:17.931167 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927858    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:18:17.931167 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927862    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:18:17.931167 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927867    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:18:17.931167 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927871    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:18:17.931167 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927876    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:18:17.931167 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927880    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:18:17.931167 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927884    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:18:17.931167 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927889    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:18:17.931167 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927894    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:18:17.931167 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927898    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:18:17.931167 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927902    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:18:17.931167 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927906    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:18:17.931167 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927911    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:18:17.931167 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927916    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:18:17.931167 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927921    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:18:17.931167 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927925    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:18:17.931167 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927931    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:18:17.931167 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927937    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:18:17.931167 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927944    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:18:17.931730 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927948    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:18:17.931730 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927953    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:18:17.931730 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.927957    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:18:17.931730 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929268    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:18:17.931730 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929279    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:18:17.931730 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929284    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:18:17.931730 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929288    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:18:17.931730 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929293    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:18:17.931730 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929297    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:18:17.931730 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929301    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:18:17.931730 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929306    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:18:17.931730 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929311    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:18:17.931730 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929315    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:18:17.931730 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929320    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:18:17.931730 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929324    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:18:17.931730 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929331    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:18:17.931730 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929337    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:18:17.931730 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929341    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:18:17.931730 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929345    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:18:17.932191 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929350    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:18:17.932191 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929354    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:18:17.932191 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929359    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:18:17.932191 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929363    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:18:17.932191 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929368    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:18:17.932191 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929373    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:18:17.932191 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929377    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:18:17.932191 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929382    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:18:17.932191 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929386    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:18:17.932191 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929390    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:18:17.932191 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929395    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:18:17.932191 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929399    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:18:17.932191 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929404    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:18:17.932191 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929409    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:18:17.932191 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929413    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:18:17.932191 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929418    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:18:17.932191 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929423    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:18:17.932191 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929427    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:18:17.932191 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929432    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:18:17.932191 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929436    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:18:17.932934 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929440    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:18:17.932934 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929445    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:18:17.932934 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929449    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:18:17.932934 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929454    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:18:17.932934 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929459    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:18:17.932934 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929464    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:18:17.932934 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929468    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:18:17.932934 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929473    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:18:17.932934 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929477    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:18:17.932934 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929482    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:18:17.932934 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929486    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:18:17.932934 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929490    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:18:17.932934 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929495    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:18:17.932934 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929499    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:18:17.932934 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929504    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:18:17.932934 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929510    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:18:17.932934 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929514    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:18:17.932934 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929519    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:18:17.932934 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929523    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:18:17.933768 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929528    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:18:17.933768 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929533    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:18:17.933768 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929537    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:18:17.933768 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929541    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:18:17.933768 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929546    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:18:17.933768 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929551    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:18:17.933768 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929555    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:18:17.933768 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929559    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:18:17.933768 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929564    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:18:17.933768 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929568    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:18:17.933768 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929572    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:18:17.933768 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929577    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:18:17.933768 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929581    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:18:17.933768 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929584    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:18:17.933768 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929589    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:18:17.933768 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929596    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:18:17.933768 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929602    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:18:17.933768 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929607    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:18:17.933768 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929635    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:18:17.934583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929641    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:18:17.934583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929646    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:18:17.934583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929650    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:18:17.934583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929653    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:18:17.934583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929658    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:18:17.934583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929662    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:18:17.934583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929667    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:18:17.934583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929672    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:18:17.934583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929676    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:18:17.934583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929681    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:18:17.934583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929685    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:18:17.934583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929690    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:18:17.934583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929695    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:18:17.934583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929699    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:18:17.934583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929704    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:18:17.934583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929708    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:18:17.934583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929713    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:18:17.934583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929718    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:18:17.934583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929722    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:18:17.934583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929733    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:18:17.935203 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929737    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:18:17.935203 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929742    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:18:17.935203 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929746    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:18:17.935203 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929751    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:18:17.935203 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929755    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:18:17.935203 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929759    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:18:17.935203 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929764    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:18:17.935203 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929768    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:18:17.935203 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929773    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:18:17.935203 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929779    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:18:17.935203 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929783    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:18:17.935203 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.929788    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:18:17.935203 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930695    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:18:17.935203 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930707    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:18:17.935203 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930712    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:18:17.935203 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930716    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:18:17.935203 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930720    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:18:17.935203 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930725    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:18:17.935203 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930730    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:18:17.935203 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930734    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:18:17.935808 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930739    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:18:17.935808 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930743    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:18:17.935808 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930747    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:18:17.935808 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930752    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:18:17.935808 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930756    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:18:17.935808 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930760    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:18:17.935808 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930764    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:18:17.935808 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930769    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:18:17.935808 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930776    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:18:17.935808 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930781    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:18:17.935808 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930786    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:18:17.935808 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930790    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:18:17.935808 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930795    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:18:17.935808 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930799    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:18:17.935808 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930803    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:18:17.935808 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930808    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:18:17.935808 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930812    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:18:17.935808 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930818    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:18:17.935808 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930823    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:18:17.936390 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930828    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:18:17.936390 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930833    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:18:17.936390 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930838    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:18:17.936390 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930843    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:18:17.936390 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930847    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:18:17.936390 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930852    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:18:17.936390 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930858    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:18:17.936390 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930863    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:18:17.936390 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930867    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:18:17.936390 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930872    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:18:17.936390 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930876    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:18:17.936390 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930880    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:18:17.936390 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930884    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:18:17.936390 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930889    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:18:17.936390 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930893    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:18:17.936390 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930898    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:18:17.936390 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930902    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:18:17.936390 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930907    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:18:17.936390 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930911    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:18:17.936864 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930915    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:18:17.936864 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930922    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:18:17.936864 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930926    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:18:17.936864 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930930    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:18:17.936864 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930934    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:18:17.936864 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930939    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:18:17.936864 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930943    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:18:17.936864 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930948    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:18:17.936864 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930952    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:18:17.936864 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930956    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:18:17.936864 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930960    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:18:17.936864 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930965    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:18:17.936864 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930969    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:18:17.936864 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930974    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:18:17.936864 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930978    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:18:17.936864 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930982    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:18:17.936864 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930988    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:18:17.936864 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930992    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:18:17.936864 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.930997    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:18:17.937378 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931004    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:18:17.937378 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931009    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:18:17.937378 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931013    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:18:17.937378 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931017    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:18:17.937378 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931021    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:18:17.937378 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931026    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:18:17.937378 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931029    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:18:17.937378 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931033    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:18:17.937378 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931038    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:18:17.937378 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931042    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:18:17.937378 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931046    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:18:17.937378 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931050    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:18:17.937378 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931055    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:18:17.937378 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931059    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:18:17.937378 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931069    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:18:17.937378 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931073    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:18:17.937378 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931078    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:18:17.937378 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931083    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:18:17.937378 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931088    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:18:17.937378 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931092    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:18:17.937947 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931097    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:18:17.937947 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931101    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:18:17.937947 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931105    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:18:17.937947 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931109    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:18:17.937947 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931114    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:18:17.937947 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931118    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:18:17.937947 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931122    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:18:17.937947 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931126    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:18:17.937947 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931131    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:18:17.937947 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931135    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:18:17.937947 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931140    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:18:17.937947 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931144    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:18:17.937947 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931150    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:18:17.937947 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931156    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:18:17.937947 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931161    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:18:17.937947 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931167    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:18:17.937947 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931174    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:18:17.937947 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931179    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:18:17.937947 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931183    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:18:17.938393 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931187    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:18:17.938393 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.931192    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:18:17.938393 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932406    2580 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 00:18:17.938393 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932425    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 00:18:17.938393 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932436    2580 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 00:18:17.938393 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932444    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 00:18:17.938393 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932451    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 00:18:17.938393 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932457    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 00:18:17.938393 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932465    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 00:18:17.938393 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932472    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 00:18:17.938393 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932478    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 00:18:17.938393 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932485    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 00:18:17.938393 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932491    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 00:18:17.938393 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932497    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 00:18:17.938393 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932503    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 00:18:17.938393 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932508    2580 flags.go:64] FLAG: --cgroup-root=""
Oct 31 00:18:17.938393 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932513    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 00:18:17.938393 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932518    2580 flags.go:64] FLAG: --client-ca-file=""
Oct 31 00:18:17.938393 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932523    2580 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 00:18:17.938393 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932527    2580 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 00:18:17.938393 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932535    2580 flags.go:64] FLAG: --cluster-domain=""
Oct 31 00:18:17.938393 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932540    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 00:18:17.938393 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932545    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 00:18:17.938393 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932551    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 00:18:17.939066 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932556    2580 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 00:18:17.939066 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932564    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 00:18:17.939066 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932569    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 00:18:17.939066 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932576    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 00:18:17.939066 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932581    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 00:18:17.939066 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932586    2580 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 00:18:17.939066 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932591    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 00:18:17.939066 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932596    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 00:18:17.939066 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932602    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 00:18:17.939066 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932606    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 00:18:17.939066 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932627    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 00:18:17.939066 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932632    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 00:18:17.939066 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932637    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 00:18:17.939066 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932642    2580 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 00:18:17.939066 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932646    2580 flags.go:64] FLAG: --enable-server="true"
Oct 31 00:18:17.939066 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932652    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 00:18:17.939066 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932659    2580 flags.go:64] FLAG: --event-burst="100"
Oct 31 00:18:17.939066 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932664    2580 flags.go:64] FLAG: --event-qps="50"
Oct 31 00:18:17.939066 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932669    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 00:18:17.939066 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932674    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 00:18:17.939066 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932679    2580 flags.go:64] FLAG: --eviction-hard=""
Oct 31 00:18:17.939066 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932686    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 00:18:17.939066 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932691    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 00:18:17.939066 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932697    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 00:18:17.939066 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932703    2580 flags.go:64] FLAG: --eviction-soft=""
Oct 31 00:18:17.939682 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932708    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 00:18:17.939682 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932712    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 00:18:17.939682 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932717    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 00:18:17.939682 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932722    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 00:18:17.939682 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932727    2580 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 00:18:17.939682 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932731    2580 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 00:18:17.939682 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932736    2580 flags.go:64] FLAG: --feature-gates=""
Oct 31 00:18:17.939682 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932743    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 00:18:17.939682 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932749    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 00:18:17.939682 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932755    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 00:18:17.939682 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932760    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 00:18:17.939682 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932766    2580 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 00:18:17.939682 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932771    2580 flags.go:64] FLAG: --help="false"
Oct 31 00:18:17.939682 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932776    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-128-253.ec2.internal"
Oct 31 00:18:17.939682 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932781    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 00:18:17.939682 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932786    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 00:18:17.939682 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932791    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 00:18:17.939682 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932797    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 00:18:17.939682 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932803    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 00:18:17.939682 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932808    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 00:18:17.939682 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932812    2580 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 00:18:17.939682 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932817    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 00:18:17.939682 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932822    2580 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 00:18:17.940220 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932827    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 00:18:17.940220 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932832    2580 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 00:18:17.940220 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932836    2580 flags.go:64] FLAG: --kube-reserved=""
Oct 31 00:18:17.940220 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932841    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 00:18:17.940220 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932846    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 00:18:17.940220 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932851    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 00:18:17.940220 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932856    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 00:18:17.940220 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932860    2580 flags.go:64] FLAG: --lock-file=""
Oct 31 00:18:17.940220 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932865    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 00:18:17.940220 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932870    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 00:18:17.940220 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932875    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 00:18:17.940220 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932884    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 00:18:17.940220 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932889    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 00:18:17.940220 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932894    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 00:18:17.940220 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932899    2580 flags.go:64] FLAG: --logging-format="text"
Oct 31 00:18:17.940220 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932904    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 00:18:17.940220 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932909    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 00:18:17.940220 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932914    2580 flags.go:64] FLAG: --manifest-url=""
Oct 31 00:18:17.940220 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932919    2580 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 00:18:17.940220 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932926    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 00:18:17.940220 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932931    2580 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 00:18:17.940220 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932938    2580 flags.go:64] FLAG: --max-pods="110"
Oct 31 00:18:17.940220 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932943    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 00:18:17.940220 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932947    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 00:18:17.940220 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932952    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 00:18:17.940889 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932956    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 00:18:17.940889 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932962    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 00:18:17.940889 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932967    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 00:18:17.940889 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932972    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 00:18:17.940889 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932984    2580 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 00:18:17.940889 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932989    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 00:18:17.940889 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932994    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 00:18:17.940889 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.932999    2580 flags.go:64] FLAG: --pod-cidr=""
Oct 31 00:18:17.940889 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933004    2580 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 00:18:17.940889 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933009    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 00:18:17.940889 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933014    2580 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 00:18:17.940889 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933018    2580 flags.go:64] FLAG: --port="10250"
Oct 31 00:18:17.940889 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933023    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 00:18:17.940889 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933029    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05ef2293b9399d368"
Oct 31 00:18:17.940889 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933034    2580 flags.go:64] FLAG: --qos-reserved=""
Oct 31 00:18:17.940889 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933038    2580 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 00:18:17.940889 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933043    2580 flags.go:64] FLAG: --register-node="true"
Oct 31 00:18:17.940889 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933048    2580 flags.go:64] FLAG: --register-with-taints=""
Oct 31 00:18:17.940889 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933055    2580 flags.go:64] FLAG: --registry-burst="10"
Oct 31 00:18:17.940889 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933060    2580 flags.go:64] FLAG: --registry-qps="5"
Oct 31 00:18:17.940889 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933065    2580 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 00:18:17.940889 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933069    2580 flags.go:64] FLAG: --reserved-memory=""
Oct 31 00:18:17.940889 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933076    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 00:18:17.940889 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933080    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 00:18:17.940889 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933086    2580 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 00:18:17.940889 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933091    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 00:18:17.941490 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933096    2580 flags.go:64] FLAG: --runonce="false"
Oct 31 00:18:17.941490 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933100    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 00:18:17.941490 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933105    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 00:18:17.941490 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933115    2580 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 00:18:17.941490 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933120    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 00:18:17.941490 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933125    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 00:18:17.941490 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933130    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 00:18:17.941490 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933136    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 00:18:17.941490 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933141    2580 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 00:18:17.941490 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933145    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 00:18:17.941490 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933153    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 00:18:17.941490 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933158    2580 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 00:18:17.941490 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933162    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 00:18:17.941490 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933168    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 00:18:17.941490 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933173    2580 flags.go:64] FLAG: --system-cgroups=""
Oct 31 00:18:17.941490 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933177    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 00:18:17.941490 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933186    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 00:18:17.941490 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933191    2580 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 00:18:17.941490 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933196    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 00:18:17.941490 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933203    2580 flags.go:64] FLAG: --tls-min-version=""
Oct 31 00:18:17.941490 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933208    2580 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 00:18:17.941490 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933212    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 00:18:17.941490 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933217    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 00:18:17.941490 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933222    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 00:18:17.941490 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933227    2580 flags.go:64] FLAG: --v="2"
Oct 31 00:18:17.942107 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933234    2580 flags.go:64] FLAG: --version="false"
Oct 31 00:18:17.942107 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933241    2580 flags.go:64] FLAG: --vmodule=""
Oct 31 00:18:17.942107 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933247    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 00:18:17.942107 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.933252    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 00:18:17.942107 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933495    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:18:17.942107 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933503    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:18:17.942107 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933508    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:18:17.942107 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933513    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:18:17.942107 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933520    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:18:17.942107 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933527    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:18:17.942107 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933533    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:18:17.942107 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933547    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:18:17.942107 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933552    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:18:17.942107 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933556    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:18:17.942107 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933561    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:18:17.942107 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933565    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:18:17.942107 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933569    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:18:17.942107 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933576    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:18:17.942107 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933580    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:18:17.942107 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933585    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:18:17.942568 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933589    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:18:17.942568 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933593    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:18:17.942568 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933597    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:18:17.942568 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933602    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:18:17.942568 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933606    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:18:17.942568 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933626    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:18:17.942568 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933632    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:18:17.942568 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933636    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:18:17.942568 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933640    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:18:17.942568 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933644    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:18:17.942568 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933649    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:18:17.942568 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933653    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:18:17.942568 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933658    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:18:17.942568 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933662    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:18:17.942568 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933666    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:18:17.942568 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933671    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:18:17.942568 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933675    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:18:17.942568 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933681    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:18:17.942568 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933686    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:18:17.943072 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933691    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:18:17.943072 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933695    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:18:17.943072 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933700    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:18:17.943072 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933704    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:18:17.943072 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933708    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:18:17.943072 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933720    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:18:17.943072 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933732    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:18:17.943072 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933737    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:18:17.943072 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933742    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:18:17.943072 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933746    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:18:17.943072 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933752    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:18:17.943072 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933756    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:18:17.943072 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933763    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:18:17.943072 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933768    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:18:17.943072 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933773    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:18:17.943072 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933777    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:18:17.943072 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933781    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:18:17.943072 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933786    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:18:17.943072 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933790    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:18:17.943072 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933795    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:18:17.943644 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933799    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:18:17.943644 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933804    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:18:17.943644 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933808    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:18:17.943644 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933812    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:18:17.943644 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933817    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:18:17.943644 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933822    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:18:17.943644 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933826    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:18:17.943644 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933830    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:18:17.943644 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933834    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:18:17.943644 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933839    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:18:17.943644 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933846    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:18:17.943644 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933851    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:18:17.943644 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933855    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:18:17.943644 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933860    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:18:17.943644 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933864    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:18:17.943644 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933869    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:18:17.943644 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933873    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:18:17.943644 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933878    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:18:17.943644 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933894    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:18:17.943644 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933899    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:18:17.944134 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933904    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:18:17.944134 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933908    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:18:17.944134 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933916    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:18:17.944134 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933921    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:18:17.944134 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933925    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:18:17.944134 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933930    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:18:17.944134 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933935    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:18:17.944134 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933939    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:18:17.944134 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933943    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:18:17.944134 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933948    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:18:17.944134 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933952    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:18:17.944134 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933957    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:18:17.944134 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933961    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:18:17.944134 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933965    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:18:17.944134 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933969    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:18:17.944134 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933974    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:18:17.944134 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933978    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:18:17.944134 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933983    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:18:17.944134 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933987    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:18:17.944134 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933991    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:18:17.944601 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.933996    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:18:17.944601 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934000    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:18:17.944601 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934007    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:18:17.944601 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934012    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:18:17.944601 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934016    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:18:17.944601 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934020    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:18:17.944601 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934025    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:18:17.944601 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934029    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:18:17.944601 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934034    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:18:17.944601 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934039    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:18:17.944601 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934043    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:18:17.944601 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.934100    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:18:17.944601 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934168    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:18:17.944601 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934176    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:18:17.944601 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934180    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:18:17.944601 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934186    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:18:17.945006 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934191    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:18:17.945006 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934195    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:18:17.945006 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934199    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:18:17.945006 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934203    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:18:17.945006 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934208    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:18:17.945006 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934212    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:18:17.945006 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934217    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:18:17.945006 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934221    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:18:17.945006 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934225    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:18:17.945006 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934230    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:18:17.945006 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934234    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:18:17.945006 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934238    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:18:17.945006 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934242    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:18:17.945006 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934247    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:18:17.945006 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934251    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:18:17.945006 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934256    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:18:17.945006 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934260    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:18:17.945006 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934264    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:18:17.945006 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934270    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:18:17.945455 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934275    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:18:17.945455 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934279    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:18:17.945455 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934283    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:18:17.945455 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934288    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:18:17.945455 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934295    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:18:17.945455 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934301    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:18:17.945455 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934306    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:18:17.945455 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934310    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:18:17.945455 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934314    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:18:17.945455 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934326    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:18:17.945455 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934332    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:18:17.945455 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934336    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:18:17.945455 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934342    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:18:17.945455 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934347    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:18:17.945455 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934351    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:18:17.945455 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934355    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:18:17.945455 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934360    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:18:17.945455 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934365    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:18:17.945455 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934369    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:18:17.945938 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934373    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:18:17.945938 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934378    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:18:17.945938 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934382    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:18:17.945938 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934387    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:18:17.945938 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934391    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:18:17.945938 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934395    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:18:17.945938 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934399    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:18:17.945938 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934404    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:18:17.945938 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934408    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:18:17.945938 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934412    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:18:17.945938 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934416    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:18:17.945938 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934421    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:18:17.945938 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934427    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:18:17.945938 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934431    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:18:17.945938 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934436    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:18:17.945938 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934440    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:18:17.945938 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934444    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:18:17.945938 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934449    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:18:17.945938 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934453    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:18:17.946429 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934457    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:18:17.946429 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934461    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:18:17.946429 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934466    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:18:17.946429 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934470    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:18:17.946429 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934482    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:18:17.946429 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934486    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:18:17.946429 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934492    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:18:17.946429 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934496    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:18:17.946429 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934500    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:18:17.946429 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934507    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:18:17.946429 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934514    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:18:17.946429 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934519    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:18:17.946429 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934527    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:18:17.946429 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934532    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:18:17.946429 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934536    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:18:17.946429 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934549    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:18:17.946429 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934555    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:18:17.946429 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934559    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:18:17.946429 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934563    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:18:17.946429 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934568    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:18:17.946921 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934572    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:18:17.946921 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934577    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:18:17.946921 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934581    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:18:17.946921 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934585    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:18:17.946921 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934590    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:18:17.946921 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934595    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:18:17.946921 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934599    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:18:17.946921 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934603    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:18:17.946921 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934608    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:18:17.946921 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934630    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:18:17.946921 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934635    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:18:17.946921 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934639    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:18:17.946921 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934643    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:18:17.946921 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934647    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:18:17.946921 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934652    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:18:17.946921 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934657    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:18:17.946921 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934661    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:18:17.946921 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934674    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:18:17.946921 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934680    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:18:17.946921 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934685    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:18:17.947382 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934689    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:18:17.947382 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934694    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:18:17.947382 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934700    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:18:17.947382 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934704    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:18:17.947382 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.934708    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:18:17.947382 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.942870    2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 00:18:17.947382 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.942887    2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 00:18:17.947382 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.942904    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 00:18:17.947382 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.942909    2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 00:18:17.947382 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.942978    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:18:17.947382 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.942984    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:18:17.947382 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.942987    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:18:17.947382 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.942990    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:18:17.947382 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.942993    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:18:17.947382 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.942996    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:18:17.947382 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.942999    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:18:17.947382 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943002    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:18:17.947382 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943005    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:18:17.947382 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943007    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:18:17.947901 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943010    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:18:17.947901 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943013    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:18:17.947901 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943015    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:18:17.947901 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943018    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:18:17.947901 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943020    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:18:17.947901 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943023    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:18:17.947901 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943025    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:18:17.947901 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943028    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:18:17.947901 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943030    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:18:17.947901 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943033    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:18:17.947901 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943035    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:18:17.947901 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943038    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:18:17.947901 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943041    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:18:17.947901 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943043    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:18:17.947901 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943046    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:18:17.947901 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943049    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:18:17.947901 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943051    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:18:17.947901 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943054    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:18:17.947901 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943057    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:18:17.948336 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943066    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:18:17.948336 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943069    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:18:17.948336 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943072    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:18:17.948336 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943075    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:18:17.948336 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943078    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:18:17.948336 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943081    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:18:17.948336 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943084    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:18:17.948336 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943087    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:18:17.948336 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943091    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:18:17.948336 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943096    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:18:17.948336 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943100    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:18:17.948336 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943103    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:18:17.948336 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943106    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:18:17.948336 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943109    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:18:17.948336 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943111    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:18:17.948336 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943114    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:18:17.948336 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943117    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:18:17.948336 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943120    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:18:17.948783 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943122    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:18:17.948783 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943125    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:18:17.948783 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943128    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:18:17.948783 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943130    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:18:17.948783 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943133    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:18:17.948783 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943136    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:18:17.948783 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943138    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:18:17.948783 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943141    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:18:17.948783 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943144    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:18:17.948783 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943147    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:18:17.948783 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943150    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:18:17.948783 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943152    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:18:17.948783 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943155    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:18:17.948783 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943158    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:18:17.948783 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943161    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:18:17.948783 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943169    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:18:17.948783 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943172    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:18:17.948783 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943175    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:18:17.948783 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943178    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:18:17.948783 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943181    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:18:17.949289 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943183    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:18:17.949289 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943186    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:18:17.949289 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943189    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:18:17.949289 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943192    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:18:17.949289 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943194    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:18:17.949289 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943197    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:18:17.949289 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943200    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:18:17.949289 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943202    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:18:17.949289 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943205    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:18:17.949289 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943208    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:18:17.949289 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943211    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:18:17.949289 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943214    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:18:17.949289 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943217    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:18:17.949289 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943220    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:18:17.949289 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943222    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:18:17.949289 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943225    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:18:17.949289 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943228    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:18:17.949289 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943230    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:18:17.949289 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943233    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:18:17.949755 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943236    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:18:17.949755 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943238    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:18:17.949755 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943241    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:18:17.949755 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943244    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:18:17.949755 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943247    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:18:17.949755 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943249    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:18:17.949755 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943252    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:18:17.949755 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943255    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:18:17.949755 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943257    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:18:17.949755 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943265    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:18:17.949755 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943269    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:18:17.949755 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943271    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:18:17.949755 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943274    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:18:17.949755 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943277    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:18:17.949755 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943280    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:18:17.949755 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943283    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:18:17.949755 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943285    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:18:17.949755 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943288    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:18:17.949755 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943291    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:18:17.949755 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943294    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:18:17.950354 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.943323    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:18:17.950354 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943517    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:18:17.950354 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943522    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:18:17.950354 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943525    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:18:17.950354 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943528    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:18:17.950354 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943531    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:18:17.950354 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943534    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:18:17.950354 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943536    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:18:17.950354 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943539    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:18:17.950354 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943542    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:18:17.950354 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943544    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:18:17.950354 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943547    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:18:17.950354 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943550    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:18:17.950354 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943552    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:18:17.950354 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943555    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:18:17.950354 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943558    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:18:17.950760 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943560    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:18:17.950760 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943563    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:18:17.950760 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943566    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:18:17.950760 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943569    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:18:17.950760 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943571    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:18:17.950760 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943574    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:18:17.950760 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943582    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:18:17.950760 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943585    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:18:17.950760 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943588    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:18:17.950760 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943590    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:18:17.950760 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943593    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:18:17.950760 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943596    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:18:17.950760 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943598    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:18:17.950760 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943602    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:18:17.950760 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943606    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:18:17.950760 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943609    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:18:17.950760 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943629    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:18:17.950760 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943634    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:18:17.950760 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943638    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:18:17.950760 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943640    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:18:17.951290 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943643    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:18:17.951290 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943646    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:18:17.951290 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943648    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:18:17.951290 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943651    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:18:17.951290 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943654    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:18:17.951290 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943657    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:18:17.951290 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943660    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:18:17.951290 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943663    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:18:17.951290 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943665    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:18:17.951290 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943668    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:18:17.951290 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943670    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:18:17.951290 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943673    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:18:17.951290 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943676    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:18:17.951290 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943678    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:18:17.951290 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943681    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:18:17.951290 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943684    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:18:17.951290 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943687    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:18:17.951290 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943690    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:18:17.951290 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943692    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:18:17.951880 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943706    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:18:17.951880 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943709    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:18:17.951880 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943711    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:18:17.951880 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943714    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:18:17.951880 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943717    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:18:17.951880 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943719    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:18:17.951880 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943722    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:18:17.951880 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943724    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:18:17.951880 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943727    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:18:17.951880 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943729    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:18:17.951880 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943732    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:18:17.951880 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943734    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:18:17.951880 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943737    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:18:17.951880 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943740    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:18:17.951880 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943742    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:18:17.951880 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943753    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:18:17.951880 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943756    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:18:17.951880 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943758    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:18:17.951880 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943761    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:18:17.952583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943763    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:18:17.952583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943766    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:18:17.952583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943768    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:18:17.952583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943771    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:18:17.952583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943773    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:18:17.952583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943776    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:18:17.952583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943779    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:18:17.952583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943782    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:18:17.952583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943786    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:18:17.952583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943790    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:18:17.952583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943793    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:18:17.952583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943797    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:18:17.952583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943799    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:18:17.952583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943802    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:18:17.952583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943810    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:18:17.952583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943813    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:18:17.952583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943816    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:18:17.952583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943819    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:18:17.952583 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943821    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:18:17.953312 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943824    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:18:17.953312 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943826    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:18:17.953312 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943829    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:18:17.953312 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943832    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:18:17.953312 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943834    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:18:17.953312 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943837    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:18:17.953312 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943840    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:18:17.953312 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943842    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:18:17.953312 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943845    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:18:17.953312 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943847    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:18:17.953312 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943850    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:18:17.953312 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943853    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:18:17.953312 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943855    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:18:17.953312 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943858    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:18:17.953312 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.943882    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:18:17.953312 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943919    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:18:17.954008 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943924    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:18:17.954008 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943927    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:18:17.954008 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943930    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:18:17.954008 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943932    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:18:17.954008 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943936    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:18:17.954008 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943942    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:18:17.954008 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943945    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:18:17.954008 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943948    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:18:17.954008 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943950    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:18:17.954008 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943953    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:18:17.954008 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943955    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:18:17.954008 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943958    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:18:17.954008 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943967    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:18:17.954008 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943970    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:18:17.954008 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943973    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:18:17.954008 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943975    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:18:17.954008 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943978    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:18:17.954008 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943980    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:18:17.954008 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943983    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:18:17.954721 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943987    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:18:17.954721 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943991    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:18:17.954721 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943994    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:18:17.954721 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.943997    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:18:17.954721 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944000    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:18:17.954721 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944003    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:18:17.954721 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944005    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:18:17.954721 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944008    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:18:17.954721 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944010    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:18:17.954721 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944013    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:18:17.954721 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944016    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:18:17.954721 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944018    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:18:17.954721 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944021    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:18:17.954721 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944023    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:18:17.954721 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944026    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:18:17.954721 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944028    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:18:17.954721 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944031    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:18:17.954721 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944034    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:18:17.954721 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944038    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:18:17.955170 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944041    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:18:17.955170 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944044    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:18:17.955170 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944046    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:18:17.955170 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944049    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:18:17.955170 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944052    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:18:17.955170 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944054    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:18:17.955170 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944057    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:18:17.955170 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944065    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:18:17.955170 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944068    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:18:17.955170 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944071    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:18:17.955170 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944073    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:18:17.955170 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944076    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:18:17.955170 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944079    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:18:17.955170 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944081    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:18:17.955170 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944084    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:18:17.955170 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944086    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:18:17.955170 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944089    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:18:17.955170 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944091    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:18:17.955170 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944094    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:18:17.955170 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944097    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:18:17.956068 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944099    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:18:17.956068 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944102    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:18:17.956068 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944104    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:18:17.956068 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944107    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:18:17.956068 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944109    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:18:17.956068 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944112    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:18:17.956068 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944114    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:18:17.956068 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944117    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:18:17.956068 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944120    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:18:17.956068 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944122    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:18:17.956068 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944125    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:18:17.956068 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944129    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:18:17.956068 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944131    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:18:17.956068 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944134    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:18:17.956068 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944138    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:18:17.956068 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944142    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:18:17.956068 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944145    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:18:17.956068 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944148    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:18:17.956068 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944151    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:18:17.956572 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944153    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:18:17.956572 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944161    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:18:17.956572 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944164    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:18:17.956572 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944167    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:18:17.956572 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944169    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:18:17.956572 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944172    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:18:17.956572 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944175    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:18:17.956572 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944177    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:18:17.956572 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944180    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:18:17.956572 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944182    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:18:17.956572 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944185    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:18:17.956572 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944188    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:18:17.956572 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944190    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:18:17.956572 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944193    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:18:17.956572 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944195    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:18:17.956572 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944198    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:18:17.956572 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944200    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:18:17.956572 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944203    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:18:17.956572 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944205    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:18:17.957038 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944208    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:18:17.957038 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944213    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:18:17.957038 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944215    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:18:17.957038 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944218    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:18:17.957038 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944222    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:18:17.957038 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944225    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:18:17.957038 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944228    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:18:17.957038 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944230    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:18:17.957038 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:17.944233    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:18:17.957038 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.944293    2580 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 00:18:17.957038 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.948700    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 00:18:17.957038 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.949853    2580 server.go:1014] "Starting client certificate rotation"
Oct 31 00:18:17.957038 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.949960    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 00:18:17.957038 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.950015    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 00:18:17.957038 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.954365    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 00:18:17.957038 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.955495    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 00:18:17.967102 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.967079    2580 log.go:25] "Validated CRI v1 runtime API"
Oct 31 00:18:17.971237 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.971221    2580 log.go:25] "Validated CRI v1 image API"
Oct 31 00:18:17.972075 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.972051    2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 00:18:17.975040 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.975018    2580 fs.go:135] Filesystem UUIDs: map[778d9087-c106-4487-91b0-5334878adcc3:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 88d3d325-1b23-43d0-b4d6-8af4f06ba1cf:/dev/nvme0n1p4]
Oct 31 00:18:17.975105 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.975039    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 00:18:17.980433 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.980324    2580 manager.go:217] Machine: {Timestamp:2026-10-31 00:18:17.979823233 +0000 UTC m=+0.378285084 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102257 MemoryCapacity:33164345344 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec257c5ed16ec91d2717d233f5153df8 SystemUUID:ec257c5e-d16e-c91d-2717-d233f5153df8 BootID:beefdae8-801c-4b93-b9a9-7786b8dd848a Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582172672 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ed:bb:92:73 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ed:bb:92:73 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:f0:76:2d:93:ef Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164345344 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 00:18:17.980433 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.980430    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 00:18:17.980568 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.980552    2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 00:18:17.984688 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.984651    2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 00:18:17.984848 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.984689    2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-253.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 00:18:17.984892 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.984851    2580 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 00:18:17.984892 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.984860    2580 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 00:18:17.984892 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.984888    2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 00:18:17.984974 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.984902    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 00:18:17.986185 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.986172    2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 00:18:17.989916 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.989901    2580 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 00:18:17.990057 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.990047    2580 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 00:18:17.990865 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.990854    2580 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 00:18:17.990922 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.990877    2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 00:18:17.990922 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.990892    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 00:18:17.991007 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.990956    2580 kubelet.go:405] "Adding apiserver pod source"
Oct 31 00:18:17.991007 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.990995    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 00:18:17.991493 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.991475    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 00:18:17.994139 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:17.994118    2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 00:18:18.001498 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.001476    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 00:18:18.002419 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.002401    2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 00:18:18.002525 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.002442    2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 00:18:18.002667 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.002653    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 00:18:18.002725 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.002673    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 00:18:18.002725 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.002683    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 00:18:18.002725 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.002691    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 00:18:18.002725 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.002700    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 00:18:18.002725 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.002709    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 00:18:18.003332 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.003321    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 00:18:18.003381 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.003336    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 00:18:18.003381 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.003349    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 00:18:18.003381 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.003359    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 00:18:18.003381 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.003373    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 00:18:18.003523 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.003387    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 00:18:18.004319 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.004309    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 00:18:18.004371 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.004325    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 00:18:18.017109 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.017091    2580 server.go:1263] "Started kubelet"
Oct 31 00:18:18.017206 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.017155    2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 00:18:18.017276 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.017186    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 00:18:18.017317 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.017305    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 00:18:18.017962 ip-10-0-128-253 systemd[1]: Started Kubernetes Kubelet.
Oct 31 00:18:18.018773 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.018755    2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 00:18:18.019404 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.019377    2580 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 00:18:18.023220 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:18.023196    2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 00:18:18.027406 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.027380    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 00:18:18.028085 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.028071    2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 00:18:18.028164 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:18.028145    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-253.ec2.internal\" not found"
Oct 31 00:18:18.028872 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.028853    2580 factory.go:55] Registering systemd factory
Oct 31 00:18:18.028959 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.028878    2580 factory.go:223] Registration of the systemd container factory successfully
Oct 31 00:18:18.029011 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.028989    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 00:18:18.029011 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.028990    2580 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 00:18:18.029090 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.029014    2580 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 00:18:18.029138 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.029126    2580 factory.go:153] Registering CRI-O factory
Oct 31 00:18:18.029178 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.029139    2580 factory.go:223] Registration of the crio container factory successfully
Oct 31 00:18:18.029178 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.029139    2580 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 00:18:18.029178 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.029151    2580 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 00:18:18.029262 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.029185    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 00:18:18.029262 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.029199    2580 factory.go:103] Registering Raw factory
Oct 31 00:18:18.029262 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.029210    2580 manager.go:1199] Started watching for new ooms in manager
Oct 31 00:18:18.030207 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.030186    2580 manager.go:319] Starting recovery of all containers
Oct 31 00:18:18.037839 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.037807    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-253.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 00:18:18.041326 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:18.037805    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-253.ec2.internal.18e375299165b928  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-253.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-253.ec2.internal,},FirstTimestamp:2026-10-31 00:18:18.01706116 +0000 UTC m=+0.415523012,LastTimestamp:2026-10-31 00:18:18.01706116 +0000 UTC m=+0.415523012,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-253.ec2.internal,}"
Oct 31 00:18:18.045969 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.045945    2580 manager.go:324] Recovery completed
Oct 31 00:18:18.050933 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.050916    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:18:18.051686 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.051664    2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 00:18:18.051762 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.051747    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q7d9m"
Oct 31 00:18:18.053556 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.053542    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientMemory"
Oct 31 00:18:18.053640 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.053575    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasNoDiskPressure"
Oct 31 00:18:18.053640 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.053589    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientPID"
Oct 31 00:18:18.054247 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.054231    2580 cpu_manager.go:226] "Starting" policy="none"
Oct 31 00:18:18.054247 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.054244    2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 00:18:18.054340 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.054247    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:18:18.054340 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.054266    2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 00:18:18.055033 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.055015    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientMemory"
Oct 31 00:18:18.055109 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.055044    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasNoDiskPressure"
Oct 31 00:18:18.055109 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.055054    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientPID"
Oct 31 00:18:18.055503 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.055492    2580 policy_none.go:50] "Start"
Oct 31 00:18:18.055537 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.055509    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 00:18:18.055537 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.055519    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 00:18:18.056501 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.056488    2580 policy_none.go:44] "Start"
Oct 31 00:18:18.057734 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.057710    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q7d9m"
Oct 31 00:18:18.059359 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:18.059330    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 00:18:18.059439 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:18.059390    2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-253.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 00:18:18.059588 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:18.059550    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-253.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 00:18:18.091917 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.091896    2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:18:18.109376 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.092388    2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 00:18:18.109376 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.095573    2580 manager.go:341] "Starting Device Plugin manager"
Oct 31 00:18:18.109376 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:18.095601    2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 00:18:18.109376 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.095611    2580 server.go:87] "Starting device plugin registration server"
Oct 31 00:18:18.109376 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.095859    2580 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 00:18:18.109376 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.095872    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 00:18:18.109376 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.095970    2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 00:18:18.109376 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.096095    2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 00:18:18.109376 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.096102    2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 00:18:18.109376 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:18.096974    2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 00:18:18.109376 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:18.097217    2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-253.ec2.internal\" not found"
Oct 31 00:18:18.112703 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.109633    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 00:18:18.112703 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.110750    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 00:18:18.112703 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.111392    2580 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 00:18:18.112703 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.111425    2580 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 00:18:18.112703 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:18.111457    2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 00:18:18.126085 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.126063    2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:18:18.196431 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.196338    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:18:18.197750 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.197722    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientMemory"
Oct 31 00:18:18.197838 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.197765    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasNoDiskPressure"
Oct 31 00:18:18.197838 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.197776    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientPID"
Oct 31 00:18:18.197838 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.197810    2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-253.ec2.internal"
Oct 31 00:18:18.211792 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.211750    2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal"]
Oct 31 00:18:18.211879 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.211853    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:18:18.212834 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.212818    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientMemory"
Oct 31 00:18:18.212925 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.212850    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasNoDiskPressure"
Oct 31 00:18:18.212925 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.212863    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientPID"
Oct 31 00:18:18.215322 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.215308    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:18:18.215471 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.215456    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Oct 31 00:18:18.215517 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.215488    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:18:18.216173 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.216158    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientMemory"
Oct 31 00:18:18.216270 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.216189    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasNoDiskPressure"
Oct 31 00:18:18.216270 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.216203    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientPID"
Oct 31 00:18:18.216270 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.216157    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientMemory"
Oct 31 00:18:18.216270 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.216262    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasNoDiskPressure"
Oct 31 00:18:18.216448 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.216274    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientPID"
Oct 31 00:18:18.217457 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.217440    2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-253.ec2.internal"
Oct 31 00:18:18.217558 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:18.217466    2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-253.ec2.internal\": node \"ip-10-0-128-253.ec2.internal\" not found"
Oct 31 00:18:18.219294 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.219276    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal"
Oct 31 00:18:18.219362 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.219311    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:18:18.220097 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.220079    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientMemory"
Oct 31 00:18:18.220168 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.220108    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasNoDiskPressure"
Oct 31 00:18:18.220168 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.220121    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientPID"
Oct 31 00:18:18.230296 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.230149    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c35a4f872afa19338c317775c3e775c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal\" (UID: \"c35a4f872afa19338c317775c3e775c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Oct 31 00:18:18.230376 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.230306    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c35a4f872afa19338c317775c3e775c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal\" (UID: \"c35a4f872afa19338c317775c3e775c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Oct 31 00:18:18.230376 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.230331    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dd74f5dcc55b4e9100e01ca71d574990-config\") pod \"kube-apiserver-proxy-ip-10-0-128-253.ec2.internal\" (UID: \"dd74f5dcc55b4e9100e01ca71d574990\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal"
Oct 31 00:18:18.249407 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:18.249386    2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-253.ec2.internal\" not found" node="ip-10-0-128-253.ec2.internal"
Oct 31 00:18:18.250096 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:18.250083    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-253.ec2.internal\" not found"
Oct 31 00:18:18.251986 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:18.251971    2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-253.ec2.internal\" not found" node="ip-10-0-128-253.ec2.internal"
Oct 31 00:18:18.331448 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.331419    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c35a4f872afa19338c317775c3e775c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal\" (UID: \"c35a4f872afa19338c317775c3e775c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Oct 31 00:18:18.331608 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.331451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c35a4f872afa19338c317775c3e775c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal\" (UID: \"c35a4f872afa19338c317775c3e775c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Oct 31 00:18:18.331608 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.331492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c35a4f872afa19338c317775c3e775c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal\" (UID: \"c35a4f872afa19338c317775c3e775c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Oct 31 00:18:18.331608 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.331512    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dd74f5dcc55b4e9100e01ca71d574990-config\") pod \"kube-apiserver-proxy-ip-10-0-128-253.ec2.internal\" (UID: \"dd74f5dcc55b4e9100e01ca71d574990\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal"
Oct 31 00:18:18.331608 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.331537    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dd74f5dcc55b4e9100e01ca71d574990-config\") pod \"kube-apiserver-proxy-ip-10-0-128-253.ec2.internal\" (UID: \"dd74f5dcc55b4e9100e01ca71d574990\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal"
Oct 31 00:18:18.331608 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.331568    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c35a4f872afa19338c317775c3e775c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal\" (UID: \"c35a4f872afa19338c317775c3e775c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Oct 31 00:18:18.351170 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:18.351142    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-253.ec2.internal\" not found"
Oct 31 00:18:18.451801 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:18.451740    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-253.ec2.internal\" not found"
Oct 31 00:18:18.552163 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:18.552125    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-253.ec2.internal\" not found"
Oct 31 00:18:18.554299 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.554280    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Oct 31 00:18:18.557433 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:18.557417    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal"
Oct 31 00:18:18.653303 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:18.653260    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-253.ec2.internal\" not found"
Oct 31 00:18:19.839953 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:19.839874    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-253.ec2.internal\" not found"
Oct 31 00:18:19.940460 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:19.940417    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-253.ec2.internal\" not found"
Oct 31 00:18:20.035974 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:20.035939    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 00:18:20.036445 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:20.036073    2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 00:18:20.036445 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:20.036107    2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 00:18:20.041126 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:20.041092    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-253.ec2.internal\" not found"
Oct 31 00:18:20.118629 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:20.118549    2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:18:20.150008 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:20.149967    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:13:19 +0000 UTC" deadline="2028-05-18 17:39:03.661386909 +0000 UTC"
Oct 31 00:18:20.150008 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:20.150006    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13577h20m43.51138877s"
Oct 31 00:18:20.166696 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:20.166652    2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:18:20.215002 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:20.214971    2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Oct 31 00:18:20.241048 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:20.241026    2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:18:20.242099 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:20.242084    2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal"
Oct 31 00:18:20.258407 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:20.258386    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:18:20.265718 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:20.265703    2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:18:21.078086 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.078025    2580 apiserver.go:52] "Watching apiserver"
Oct 31 00:18:21.081359 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.081334    2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:18:21.115246 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.115215    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 00:18:21.124480 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.124450    2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 00:18:21.124865 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.124830    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-g24ff","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g","openshift-multus/network-metrics-daemon-5tw4x","kube-system/global-pull-secret-syncer-pncsr","openshift-image-registry/node-ca-htbtd","openshift-network-diagnostics/network-check-target-vdxch","openshift-network-operator/iptables-alerter-9j7z8","openshift-cluster-node-tuning-operator/tuned-qk5kf","openshift-dns/node-resolver-9d69r","openshift-multus/multus-additional-cni-plugins-pwt7t","openshift-ovn-kubernetes/ovnkube-node-nv7t7","kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal","openshift-multus/multus-6ljg5"]
Oct 31 00:18:21.127613 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.127590    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9j7z8"
Oct 31 00:18:21.130108 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.130087    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.132204 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.132170    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:21.132302 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:21.132242    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5tw4x" podUID="3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a"
Oct 31 00:18:21.134639 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.134536    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:21.134639 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:21.134607    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pncsr" podUID="002d2da2-bb18-4667-98c8-1770b70d9985"
Oct 31 00:18:21.136919 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.136901    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-htbtd"
Oct 31 00:18:21.139104 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.139085    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:21.139197 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:21.139152    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdxch" podUID="5b9f4a03-68eb-418e-9447-ad62473a1e9a"
Oct 31 00:18:21.142644 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.142626    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9d69r"
Oct 31 00:18:21.144690 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.144657    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:18:21.144862 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.144846    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-grldd\""
Oct 31 00:18:21.144932 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.144923    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 00:18:21.144985 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.144974    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 00:18:21.145047 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.145032    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.146553 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.146533    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4w2kf\""
Oct 31 00:18:21.147248 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.146902    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:18:21.147954 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.147934    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 00:18:21.148235 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.148217    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 00:18:21.148235 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.148222    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 00:18:21.148410 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.148369    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 00:18:21.148483 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.148467    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 00:18:21.148527 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.148501    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 00:18:21.149388 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.149367    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:18:21.151752 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.151735    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g24ff"
Oct 31 00:18:21.155956 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.155933    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.177568 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.177539    2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 00:18:21.177719 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.177616    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-h7sdt"
Oct 31 00:18:21.203943 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.203146    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal" event={"ID":"dd74f5dcc55b4e9100e01ca71d574990","Type":"ContainerStarted","Data":"5baf3e2893288f4a3179df9ca4738c5072442391df2aea0617d11ba58ce57975"}
Oct 31 00:18:21.204526 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.204493    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal" event={"ID":"c35a4f872afa19338c317775c3e775c3","Type":"ContainerStarted","Data":"0bde3ead4ffd0b3a9857b98bf246d4d6ba0a513dadbbe5dbf7b13319160da364"}
Oct 31 00:18:21.216807 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.216781    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 00:18:21.230782 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.230750    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0c02c18-5891-493e-ab22-5191677b8364-env-overrides\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.230782 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.230789    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-metrics-certs\") pod \"network-metrics-daemon-5tw4x\" (UID: \"3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a\") " pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:21.230984 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.230823    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-var-lib-cni-bin\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.230984 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.230843    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-var-lib-kubelet\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.230984 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.230867    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-sys\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.230984 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.230890    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bwkn\" (UniqueName: \"kubernetes.io/projected/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-kube-api-access-7bwkn\") pod \"network-metrics-daemon-5tw4x\" (UID: \"3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a\") " pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:21.230984 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.230910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-kubelet\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.230984 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.230932    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-run-netns\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.230984 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.230955    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-var-lib-openvswitch\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.230984 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.230975    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-sysctl-d\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.231275 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.230991    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-sysctl-conf\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.231275 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231009    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fdd0807-bb6c-4867-aad0-40af2ddc7d08-host-slash\") pod \"iptables-alerter-9j7z8\" (UID: \"4fdd0807-bb6c-4867-aad0-40af2ddc7d08\") " pod="openshift-network-operator/iptables-alerter-9j7z8"
Oct 31 00:18:21.231275 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231031    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-systemd-units\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.231275 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231056    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-cni-bin\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.231275 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231083    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.231275 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231108    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0c02c18-5891-493e-ab22-5191677b8364-ovn-node-metrics-cert\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.231275 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231131    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-sysconfig\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.231275 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231158    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95f3be30-2d65-4373-9623-155c841dca11-serviceca\") pod \"node-ca-htbtd\" (UID: \"95f3be30-2d65-4373-9623-155c841dca11\") " pod="openshift-image-registry/node-ca-htbtd"
Oct 31 00:18:21.231275 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-sys-fs\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:18:21.231275 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231250    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-log-socket\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.231275 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231270    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88p6w\" (UniqueName: \"kubernetes.io/projected/b0c02c18-5891-493e-ab22-5191677b8364-kube-api-access-88p6w\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.231616 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231287    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/002d2da2-bb18-4667-98c8-1770b70d9985-kubelet-config\") pod \"global-pull-secret-syncer-pncsr\" (UID: \"002d2da2-bb18-4667-98c8-1770b70d9985\") " pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:21.231616 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231324    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-host\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.231616 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231345    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f456866-ce05-42c8-adb6-588cf5fcd8b7-tmp\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.231616 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8190e88-1887-415f-95cb-d421f3fbc08b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:18:21.231616 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231394    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-modprobe-d\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.231616 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231422    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17401e80-fe41-417f-9840-acd7a68d17e3-agent-certs\") pod \"konnectivity-agent-g24ff\" (UID: \"17401e80-fe41-417f-9840-acd7a68d17e3\") " pod="kube-system/konnectivity-agent-g24ff"
Oct 31 00:18:21.231616 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231447    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-socket-dir\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:18:21.231616 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231478    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-etc-selinux\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:18:21.231616 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8190e88-1887-415f-95cb-d421f3fbc08b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:18:21.231616 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231526    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0c02c18-5891-493e-ab22-5191677b8364-ovnkube-config\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.231616 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231541    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/002d2da2-bb18-4667-98c8-1770b70d9985-original-pull-secret\") pod \"global-pull-secret-syncer-pncsr\" (UID: \"002d2da2-bb18-4667-98c8-1770b70d9985\") " pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:21.231616 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231555    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ec6a6238-81be-4a82-b346-b4c9fe3aefa4-hosts-file\") pod \"node-resolver-9d69r\" (UID: \"ec6a6238-81be-4a82-b346-b4c9fe3aefa4\") " pod="openshift-dns/node-resolver-9d69r"
Oct 31 00:18:21.231616 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231584    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-multus-conf-dir\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.231616 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8190e88-1887-415f-95cb-d421f3fbc08b-system-cni-dir\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:18:21.231616 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231616    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-cni-netd\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.232180 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231633    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-var-lib-cni-multus\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.232180 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231657    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8190e88-1887-415f-95cb-d421f3fbc08b-cni-binary-copy\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:18:21.232180 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231702    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk4vp\" (UniqueName: \"kubernetes.io/projected/a8190e88-1887-415f-95cb-d421f3fbc08b-kube-api-access-fk4vp\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:18:21.232180 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231727    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-systemd\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.232180 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231858    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95f3be30-2d65-4373-9623-155c841dca11-host\") pod \"node-ca-htbtd\" (UID: \"95f3be30-2d65-4373-9623-155c841dca11\") " pod="openshift-image-registry/node-ca-htbtd"
Oct 31 00:18:21.232180 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231925    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz8l7\" (UniqueName: \"kubernetes.io/projected/ec6a6238-81be-4a82-b346-b4c9fe3aefa4-kube-api-access-rz8l7\") pod \"node-resolver-9d69r\" (UID: \"ec6a6238-81be-4a82-b346-b4c9fe3aefa4\") " pod="openshift-dns/node-resolver-9d69r"
Oct 31 00:18:21.232180 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231968    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-etc-kubernetes\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.232180 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.231987    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bbs6\" (UniqueName: \"kubernetes.io/projected/4fdd0807-bb6c-4867-aad0-40af2ddc7d08-kube-api-access-6bbs6\") pod \"iptables-alerter-9j7z8\" (UID: \"4fdd0807-bb6c-4867-aad0-40af2ddc7d08\") " pod="openshift-network-operator/iptables-alerter-9j7z8"
Oct 31 00:18:21.232180 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232018    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-registration-dir\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:18:21.232180 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232042    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-device-dir\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:18:21.232180 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232059    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-slash\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.232180 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232075    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-node-log\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.232180 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232123    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0c02c18-5891-493e-ab22-5191677b8364-ovnkube-script-lib\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.232180 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232145    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-multus-socket-dir-parent\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.232180 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232162    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-multus-daemon-config\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.232180 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232185    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7pbg\" (UniqueName: \"kubernetes.io/projected/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-kube-api-access-m7pbg\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.232905 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232209    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-run\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.232905 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232224    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-tuned\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.232905 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-run-systemd\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.232905 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232309    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-system-cni-dir\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.232905 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232336    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-multus-cni-dir\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.232905 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232359    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-lib-modules\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.232905 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngkbc\" (UniqueName: \"kubernetes.io/projected/5b9f4a03-68eb-418e-9447-ad62473a1e9a-kube-api-access-ngkbc\") pod \"network-check-target-vdxch\" (UID: \"5b9f4a03-68eb-418e-9447-ad62473a1e9a\") " pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:21.232905 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232414    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-run-ovn-kubernetes\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.232905 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232437    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-hostroot\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.232905 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232458    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:18:21.232905 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8190e88-1887-415f-95cb-d421f3fbc08b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:18:21.232905 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232499    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-etc-openvswitch\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.232905 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232513    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec6a6238-81be-4a82-b346-b4c9fe3aefa4-tmp-dir\") pod \"node-resolver-9d69r\" (UID: \"ec6a6238-81be-4a82-b346-b4c9fe3aefa4\") " pod="openshift-dns/node-resolver-9d69r"
Oct 31 00:18:21.232905 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232533    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-kubernetes\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.232905 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232555    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mndzv\" (UniqueName: \"kubernetes.io/projected/95f3be30-2d65-4373-9623-155c841dca11-kube-api-access-mndzv\") pod \"node-ca-htbtd\" (UID: \"95f3be30-2d65-4373-9623-155c841dca11\") " pod="openshift-image-registry/node-ca-htbtd"
Oct 31 00:18:21.232905 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232571    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8190e88-1887-415f-95cb-d421f3fbc08b-cnibin\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:18:21.233504 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232593    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-cni-binary-copy\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.233504 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232620    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-run-netns\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.233504 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232706    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17401e80-fe41-417f-9840-acd7a68d17e3-konnectivity-ca\") pod \"konnectivity-agent-g24ff\" (UID: \"17401e80-fe41-417f-9840-acd7a68d17e3\") " pod="kube-system/konnectivity-agent-g24ff"
Oct 31 00:18:21.233504 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232724    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-var-lib-kubelet\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.233504 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4fdd0807-bb6c-4867-aad0-40af2ddc7d08-iptables-alerter-script\") pod \"iptables-alerter-9j7z8\" (UID: \"4fdd0807-bb6c-4867-aad0-40af2ddc7d08\") " pod="openshift-network-operator/iptables-alerter-9j7z8"
Oct 31 00:18:21.233504 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232763    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkmrt\" (UniqueName: \"kubernetes.io/projected/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-kube-api-access-mkmrt\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:18:21.233504 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232788    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-run-openvswitch\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.233504 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232813    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/002d2da2-bb18-4667-98c8-1770b70d9985-dbus\") pod \"global-pull-secret-syncer-pncsr\" (UID: \"002d2da2-bb18-4667-98c8-1770b70d9985\") " pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:21.233504 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232834    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-cnibin\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.233504 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232850    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-run-k8s-cni-cncf-io\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.233504 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232865    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-run-multus-certs\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.233504 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232879    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-755hv\" (UniqueName: \"kubernetes.io/projected/1f456866-ce05-42c8-adb6-588cf5fcd8b7-kube-api-access-755hv\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.233504 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.232905    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-run-ovn\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.246370 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.246343    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 00:18:21.246759 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.246741    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 00:18:21.246869 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.246836    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 00:18:21.246869 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.246867    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 00:18:21.246983 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.246873    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:18:21.246983 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.246838    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9prwf\""
Oct 31 00:18:21.246983 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.246875    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 00:18:21.246983 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.246876    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 00:18:21.246983 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.246904    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 00:18:21.246983 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.246837    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 00:18:21.246983 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.246932    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4wlc7\""
Oct 31 00:18:21.246983 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.246974    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 00:18:21.246983 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.246979    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 00:18:21.247354 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.246987    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 00:18:21.247354 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.246982    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wgmpl\""
Oct 31 00:18:21.247354 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.247003    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pp79q\""
Oct 31 00:18:21.247354 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.247012    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 00:18:21.247354 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.247023    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 00:18:21.247354 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.247031    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 00:18:21.247354 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.247037    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 00:18:21.247354 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.247086    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 00:18:21.247354 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.247178    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jw4zk\""
Oct 31 00:18:21.247354 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.247215    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dqh4t\""
Oct 31 00:18:21.247354 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.247257    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 00:18:21.247354 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.247335    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rqjpt\""
Oct 31 00:18:21.261363 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.261009    2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:18:21.334155 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334063    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-var-lib-cni-multus\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.334155 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334115    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8190e88-1887-415f-95cb-d421f3fbc08b-cni-binary-copy\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:18:21.334155 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334147    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk4vp\" (UniqueName: \"kubernetes.io/projected/a8190e88-1887-415f-95cb-d421f3fbc08b-kube-api-access-fk4vp\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:18:21.334424 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334173    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-systemd\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.334424 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334197    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95f3be30-2d65-4373-9623-155c841dca11-host\") pod \"node-ca-htbtd\" (UID: \"95f3be30-2d65-4373-9623-155c841dca11\") " pod="openshift-image-registry/node-ca-htbtd"
Oct 31 00:18:21.334424 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334202    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-var-lib-cni-multus\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.334424 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz8l7\" (UniqueName: \"kubernetes.io/projected/ec6a6238-81be-4a82-b346-b4c9fe3aefa4-kube-api-access-rz8l7\") pod \"node-resolver-9d69r\" (UID: \"ec6a6238-81be-4a82-b346-b4c9fe3aefa4\") " pod="openshift-dns/node-resolver-9d69r"
Oct 31 00:18:21.334424 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334243    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-etc-kubernetes\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.334424 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334286    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bbs6\" (UniqueName: \"kubernetes.io/projected/4fdd0807-bb6c-4867-aad0-40af2ddc7d08-kube-api-access-6bbs6\") pod \"iptables-alerter-9j7z8\" (UID: \"4fdd0807-bb6c-4867-aad0-40af2ddc7d08\") " pod="openshift-network-operator/iptables-alerter-9j7z8"
Oct 31 00:18:21.334424 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334296    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-systemd\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.334424 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334312    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-registration-dir\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:18:21.334424 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334336    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-device-dir\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:18:21.334424 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334359    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-slash\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.334424 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-node-log\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.334424 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334404    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0c02c18-5891-493e-ab22-5191677b8364-ovnkube-script-lib\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.335008 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334431    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-multus-socket-dir-parent\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.335008 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334455    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-multus-daemon-config\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.335008 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7pbg\" (UniqueName: \"kubernetes.io/projected/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-kube-api-access-m7pbg\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.335008 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334511    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-run\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.335008 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-tuned\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.335008 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334570    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-run-systemd\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.335008 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334594    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-system-cni-dir\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.335008 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334617    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-multus-cni-dir\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.335008 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-lib-modules\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.335008 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334670    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngkbc\" (UniqueName: \"kubernetes.io/projected/5b9f4a03-68eb-418e-9447-ad62473a1e9a-kube-api-access-ngkbc\") pod \"network-check-target-vdxch\" (UID: \"5b9f4a03-68eb-418e-9447-ad62473a1e9a\") " pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:21.335008 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334710    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-run-ovn-kubernetes\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.335008 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334733    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-hostroot\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.335008 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334757    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:18:21.335008 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334784    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8190e88-1887-415f-95cb-d421f3fbc08b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:18:21.335008 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-etc-openvswitch\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.335008 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334832    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec6a6238-81be-4a82-b346-b4c9fe3aefa4-tmp-dir\") pod \"node-resolver-9d69r\" (UID: \"ec6a6238-81be-4a82-b346-b4c9fe3aefa4\") " pod="openshift-dns/node-resolver-9d69r"
Oct 31 00:18:21.335008 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334856    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-kubernetes\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.335811 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mndzv\" (UniqueName: \"kubernetes.io/projected/95f3be30-2d65-4373-9623-155c841dca11-kube-api-access-mndzv\") pod \"node-ca-htbtd\" (UID: \"95f3be30-2d65-4373-9623-155c841dca11\") " pod="openshift-image-registry/node-ca-htbtd"
Oct 31 00:18:21.335811 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334905    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8190e88-1887-415f-95cb-d421f3fbc08b-cnibin\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:18:21.335811 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334928    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-cni-binary-copy\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.335811 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334946    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-run\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.335811 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334994    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95f3be30-2d65-4373-9623-155c841dca11-host\") pod \"node-ca-htbtd\" (UID: \"95f3be30-2d65-4373-9623-155c841dca11\") " pod="openshift-image-registry/node-ca-htbtd"
Oct 31 00:18:21.335811 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.334968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-run-netns\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.335811 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335028    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17401e80-fe41-417f-9840-acd7a68d17e3-konnectivity-ca\") pod \"konnectivity-agent-g24ff\" (UID: \"17401e80-fe41-417f-9840-acd7a68d17e3\") " pod="kube-system/konnectivity-agent-g24ff"
Oct 31 00:18:21.335811 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335047    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-run-netns\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.335811 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335052    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-var-lib-kubelet\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.335811 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335080    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4fdd0807-bb6c-4867-aad0-40af2ddc7d08-iptables-alerter-script\") pod \"iptables-alerter-9j7z8\" (UID: \"4fdd0807-bb6c-4867-aad0-40af2ddc7d08\") " pod="openshift-network-operator/iptables-alerter-9j7z8"
Oct 31 00:18:21.335811 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335107    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkmrt\" (UniqueName: \"kubernetes.io/projected/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-kube-api-access-mkmrt\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:18:21.335811 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335132    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-run-openvswitch\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.335811 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335156    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/002d2da2-bb18-4667-98c8-1770b70d9985-dbus\") pod \"global-pull-secret-syncer-pncsr\" (UID: \"002d2da2-bb18-4667-98c8-1770b70d9985\") " pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:21.335811 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335180    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-cnibin\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.335811 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335203    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-run-k8s-cni-cncf-io\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.335811 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-run-multus-certs\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.335811 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-755hv\" (UniqueName: \"kubernetes.io/projected/1f456866-ce05-42c8-adb6-588cf5fcd8b7-kube-api-access-755hv\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.335811 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335277    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-run-ovn\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.336486 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0c02c18-5891-493e-ab22-5191677b8364-env-overrides\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.336486 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335320    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-metrics-certs\") pod \"network-metrics-daemon-5tw4x\" (UID: \"3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a\") " pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:21.336486 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-var-lib-cni-bin\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.336486 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335346    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-run-systemd\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.336486 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335368    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-var-lib-kubelet\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.336486 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335391    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-sys\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.336486 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335412    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bwkn\" (UniqueName: \"kubernetes.io/projected/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-kube-api-access-7bwkn\") pod \"network-metrics-daemon-5tw4x\" (UID: \"3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a\") " pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:21.336486 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335433    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-kubelet\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.336486 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-run-netns\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.336486 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335467    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-system-cni-dir\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.336486 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335485    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-var-lib-openvswitch\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.336486 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335512    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-sysctl-d\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.336486 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335540    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-sysctl-conf\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.336486 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335546    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-multus-cni-dir\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.336486 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335574    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fdd0807-bb6c-4867-aad0-40af2ddc7d08-host-slash\") pod \"iptables-alerter-9j7z8\" (UID: \"4fdd0807-bb6c-4867-aad0-40af2ddc7d08\") " pod="openshift-network-operator/iptables-alerter-9j7z8"
Oct 31 00:18:21.336486 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-systemd-units\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.336486 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335613    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8190e88-1887-415f-95cb-d421f3fbc08b-cni-binary-copy\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:18:21.337237 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-cni-bin\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.337237 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.337237 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335666    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-sysctl-conf\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.337237 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335672    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0c02c18-5891-493e-ab22-5191677b8364-ovn-node-metrics-cert\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.337237 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335717    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-sysconfig\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.337237 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335750    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95f3be30-2d65-4373-9623-155c841dca11-serviceca\") pod \"node-ca-htbtd\" (UID: \"95f3be30-2d65-4373-9623-155c841dca11\") " pod="openshift-image-registry/node-ca-htbtd"
Oct 31 00:18:21.337237 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-sys-fs\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:18:21.337237 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-log-socket\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.337237 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88p6w\" (UniqueName: \"kubernetes.io/projected/b0c02c18-5891-493e-ab22-5191677b8364-kube-api-access-88p6w\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.337237 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335853    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/002d2da2-bb18-4667-98c8-1770b70d9985-kubelet-config\") pod \"global-pull-secret-syncer-pncsr\" (UID: \"002d2da2-bb18-4667-98c8-1770b70d9985\") " pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:21.337237 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335875    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-host\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.337237 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f456866-ce05-42c8-adb6-588cf5fcd8b7-tmp\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.337237 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8190e88-1887-415f-95cb-d421f3fbc08b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:18:21.337237 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335942    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-modprobe-d\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.337237 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335967    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17401e80-fe41-417f-9840-acd7a68d17e3-agent-certs\") pod \"konnectivity-agent-g24ff\" (UID: \"17401e80-fe41-417f-9840-acd7a68d17e3\") " pod="kube-system/konnectivity-agent-g24ff"
Oct 31 00:18:21.337237 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335977    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec6a6238-81be-4a82-b346-b4c9fe3aefa4-tmp-dir\") pod \"node-resolver-9d69r\" (UID: \"ec6a6238-81be-4a82-b346-b4c9fe3aefa4\") " pod="openshift-dns/node-resolver-9d69r"
Oct 31 00:18:21.337237 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.335993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-socket-dir\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:18:21.338031 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.336006    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-etc-kubernetes\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.338031 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.336020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-etc-selinux\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:18:21.338031 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.336066    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8190e88-1887-415f-95cb-d421f3fbc08b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:18:21.338031 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.336093    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0c02c18-5891-493e-ab22-5191677b8364-ovnkube-config\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.338031 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.336118    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/002d2da2-bb18-4667-98c8-1770b70d9985-original-pull-secret\") pod \"global-pull-secret-syncer-pncsr\" (UID: \"002d2da2-bb18-4667-98c8-1770b70d9985\") " pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:21.338031 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.336123    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-lib-modules\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.338031 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.336145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ec6a6238-81be-4a82-b346-b4c9fe3aefa4-hosts-file\") pod \"node-resolver-9d69r\" (UID: \"ec6a6238-81be-4a82-b346-b4c9fe3aefa4\") " pod="openshift-dns/node-resolver-9d69r"
Oct 31 00:18:21.338031 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.336203    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ec6a6238-81be-4a82-b346-b4c9fe3aefa4-hosts-file\") pod \"node-resolver-9d69r\" (UID: \"ec6a6238-81be-4a82-b346-b4c9fe3aefa4\") " pod="openshift-dns/node-resolver-9d69r"
Oct 31 00:18:21.338031 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.336231    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-registration-dir\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:18:21.338031 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.336253    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-kubernetes\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.338031 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.336278    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-device-dir\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:18:21.338031 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.336316    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-slash\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.338031 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.336351    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-node-log\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.338031 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.336369    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-run-ovn-kubernetes\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.338031 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.336418    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-hostroot\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.338031 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.336466    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:18:21.338031 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.336469    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8190e88-1887-415f-95cb-d421f3fbc08b-cnibin\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:18:21.338841 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.336957    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8190e88-1887-415f-95cb-d421f3fbc08b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:18:21.338841 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.336981    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-run-multus-certs\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.338841 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.336992    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0c02c18-5891-493e-ab22-5191677b8364-ovnkube-script-lib\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.338841 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.336986    2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 00:18:21.338841 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.337020    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-etc-openvswitch\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.338841 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.337052    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-cni-binary-copy\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.338841 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.337054    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-multus-conf-dir\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.338841 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.337080    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-multus-conf-dir\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.338841 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.337104    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8190e88-1887-415f-95cb-d421f3fbc08b-system-cni-dir\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:18:21.338841 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.337125    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/002d2da2-bb18-4667-98c8-1770b70d9985-kubelet-config\") pod \"global-pull-secret-syncer-pncsr\" (UID: \"002d2da2-bb18-4667-98c8-1770b70d9985\") " pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:21.338841 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.337133    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-cni-netd\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.338841 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.337176    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-cni-bin\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.338841 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.337179    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fdd0807-bb6c-4867-aad0-40af2ddc7d08-host-slash\") pod \"iptables-alerter-9j7z8\" (UID: \"4fdd0807-bb6c-4867-aad0-40af2ddc7d08\") " pod="openshift-network-operator/iptables-alerter-9j7z8"
Oct 31 00:18:21.338841 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.337216    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-systemd-units\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.338841 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.337274    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.338841 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.337435    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-cni-netd\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.338841 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.337585    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0c02c18-5891-493e-ab22-5191677b8364-env-overrides\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.338841 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.337596    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-run-ovn\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.339664 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.337661    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8190e88-1887-415f-95cb-d421f3fbc08b-system-cni-dir\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:18:21.339664 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.337769    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17401e80-fe41-417f-9840-acd7a68d17e3-konnectivity-ca\") pod \"konnectivity-agent-g24ff\" (UID: \"17401e80-fe41-417f-9840-acd7a68d17e3\") " pod="kube-system/konnectivity-agent-g24ff"
Oct 31 00:18:21.339664 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.337831    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-host\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.339664 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.337858    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-modprobe-d\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.339664 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.337890    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8190e88-1887-415f-95cb-d421f3fbc08b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:18:21.339664 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.338064    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-multus-daemon-config\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.339664 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.338103    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-run-netns\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.339664 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.338189    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-socket-dir\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:18:21.339664 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.338236    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-etc-selinux\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:18:21.339664 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.338292    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8190e88-1887-415f-95cb-d421f3fbc08b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:18:21.339664 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.338337    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-var-lib-openvswitch\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.339664 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.338382    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-multus-socket-dir-parent\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.339664 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.338427    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-sys-fs\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:18:21.339664 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.338459    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-run-openvswitch\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.339664 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.338512    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-var-lib-kubelet\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.339664 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:21.338584    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:18:21.339664 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.338617    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95f3be30-2d65-4373-9623-155c841dca11-serviceca\") pod \"node-ca-htbtd\" (UID: \"95f3be30-2d65-4373-9623-155c841dca11\") " pod="openshift-image-registry/node-ca-htbtd"
Oct 31 00:18:21.340459 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:21.338646    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/002d2da2-bb18-4667-98c8-1770b70d9985-original-pull-secret podName:002d2da2-bb18-4667-98c8-1770b70d9985 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:21.838626358 +0000 UTC m=+3.151025627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/002d2da2-bb18-4667-98c8-1770b70d9985-original-pull-secret") pod "global-pull-secret-syncer-pncsr" (UID: "002d2da2-bb18-4667-98c8-1770b70d9985") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:18:21.340459 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.338692    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-sysconfig\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.340459 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.338708    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-var-lib-kubelet\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.340459 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.338740    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-sys\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.340459 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.338745    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-kubelet\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.340459 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:21.338813    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:18:21.340459 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:21.338859    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-metrics-certs podName:3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:21.838845067 +0000 UTC m=+3.151244330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-metrics-certs") pod "network-metrics-daemon-5tw4x" (UID: "3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:18:21.340459 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.338855    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-log-socket\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.340459 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.338903    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-var-lib-cni-bin\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.340459 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.338963    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/002d2da2-bb18-4667-98c8-1770b70d9985-dbus\") pod \"global-pull-secret-syncer-pncsr\" (UID: \"002d2da2-bb18-4667-98c8-1770b70d9985\") " pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:21.340459 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.339189    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-cnibin\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.340459 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.339247    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-run-k8s-cni-cncf-io\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.340459 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.339381    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4fdd0807-bb6c-4867-aad0-40af2ddc7d08-iptables-alerter-script\") pod \"iptables-alerter-9j7z8\" (UID: \"4fdd0807-bb6c-4867-aad0-40af2ddc7d08\") " pod="openshift-network-operator/iptables-alerter-9j7z8"
Oct 31 00:18:21.340459 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.339395    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-sysctl-d\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.340459 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.339383    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0c02c18-5891-493e-ab22-5191677b8364-ovnkube-config\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.341719 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.341692    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0c02c18-5891-493e-ab22-5191677b8364-ovn-node-metrics-cert\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.341851 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.341809    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f456866-ce05-42c8-adb6-588cf5fcd8b7-tmp\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.351225 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.351183    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17401e80-fe41-417f-9840-acd7a68d17e3-agent-certs\") pod \"konnectivity-agent-g24ff\" (UID: \"17401e80-fe41-417f-9840-acd7a68d17e3\") " pod="kube-system/konnectivity-agent-g24ff"
Oct 31 00:18:21.351370 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.351334    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-tuned\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.359004 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:21.358976    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:18:21.359004 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:21.359000    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:18:21.359176 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:21.359014    2580 projected.go:196] Error preparing data for projected volume kube-api-access-ngkbc for pod openshift-network-diagnostics/network-check-target-vdxch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:18:21.359176 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:21.359067    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b9f4a03-68eb-418e-9447-ad62473a1e9a-kube-api-access-ngkbc podName:5b9f4a03-68eb-418e-9447-ad62473a1e9a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:21.859053251 +0000 UTC m=+3.171452514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ngkbc" (UniqueName: "kubernetes.io/projected/5b9f4a03-68eb-418e-9447-ad62473a1e9a-kube-api-access-ngkbc") pod "network-check-target-vdxch" (UID: "5b9f4a03-68eb-418e-9447-ad62473a1e9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:18:21.363485 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.362841    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz8l7\" (UniqueName: \"kubernetes.io/projected/ec6a6238-81be-4a82-b346-b4c9fe3aefa4-kube-api-access-rz8l7\") pod \"node-resolver-9d69r\" (UID: \"ec6a6238-81be-4a82-b346-b4c9fe3aefa4\") " pod="openshift-dns/node-resolver-9d69r"
Oct 31 00:18:21.363485 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.363015    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mndzv\" (UniqueName: \"kubernetes.io/projected/95f3be30-2d65-4373-9623-155c841dca11-kube-api-access-mndzv\") pod \"node-ca-htbtd\" (UID: \"95f3be30-2d65-4373-9623-155c841dca11\") " pod="openshift-image-registry/node-ca-htbtd"
Oct 31 00:18:21.363485 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.363285    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bbs6\" (UniqueName: \"kubernetes.io/projected/4fdd0807-bb6c-4867-aad0-40af2ddc7d08-kube-api-access-6bbs6\") pod \"iptables-alerter-9j7z8\" (UID: \"4fdd0807-bb6c-4867-aad0-40af2ddc7d08\") " pod="openshift-network-operator/iptables-alerter-9j7z8"
Oct 31 00:18:21.368043 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.367971    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7pbg\" (UniqueName: \"kubernetes.io/projected/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-kube-api-access-m7pbg\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.372896 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.372867    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88p6w\" (UniqueName: \"kubernetes.io/projected/b0c02c18-5891-493e-ab22-5191677b8364-kube-api-access-88p6w\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.372896 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.372885    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk4vp\" (UniqueName: \"kubernetes.io/projected/a8190e88-1887-415f-95cb-d421f3fbc08b-kube-api-access-fk4vp\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:18:21.373073 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.372898    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bwkn\" (UniqueName: \"kubernetes.io/projected/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-kube-api-access-7bwkn\") pod \"network-metrics-daemon-5tw4x\" (UID: \"3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a\") " pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:21.373413 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.373387    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-755hv\" (UniqueName: \"kubernetes.io/projected/1f456866-ce05-42c8-adb6-588cf5fcd8b7-kube-api-access-755hv\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.374004 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.373984    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkmrt\" (UniqueName: \"kubernetes.io/projected/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-kube-api-access-mkmrt\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:18:21.443940 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.443908    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9j7z8"
Oct 31 00:18:21.453327 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:21.453293    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fdd0807_bb6c_4867_aad0_40af2ddc7d08.slice/crio-cf16bf06fb01a647c3bfe579684dc0d24ef6a9de7c246b2befb30a90bfd3fe58 WatchSource:0}: Error finding container cf16bf06fb01a647c3bfe579684dc0d24ef6a9de7c246b2befb30a90bfd3fe58: Status 404 returned error can't find the container with id cf16bf06fb01a647c3bfe579684dc0d24ef6a9de7c246b2befb30a90bfd3fe58
Oct 31 00:18:21.465636 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.465602    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-htbtd"
Oct 31 00:18:21.470813 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.470792    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:21.472007 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.471983    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9d69r"
Oct 31 00:18:21.478311 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.478291    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6ljg5"
Oct 31 00:18:21.487182 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:21.487155    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec6a6238_81be_4a82_b346_b4c9fe3aefa4.slice/crio-c20b6ee53b5607496b962c020108ac6e9c8847d050b4c924742840c01fb84bed WatchSource:0}: Error finding container c20b6ee53b5607496b962c020108ac6e9c8847d050b4c924742840c01fb84bed: Status 404 returned error can't find the container with id c20b6ee53b5607496b962c020108ac6e9c8847d050b4c924742840c01fb84bed
Oct 31 00:18:21.490010 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.489979    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:18:21.491517 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:21.491493    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc0fbb5_0386_40b9_aeef_3b7aa867f625.slice/crio-b73bdbdfbf94a552b042bbcfb5d528e3de51d3cf0c5f5b7ec764f58971cdc684 WatchSource:0}: Error finding container b73bdbdfbf94a552b042bbcfb5d528e3de51d3cf0c5f5b7ec764f58971cdc684: Status 404 returned error can't find the container with id b73bdbdfbf94a552b042bbcfb5d528e3de51d3cf0c5f5b7ec764f58971cdc684
Oct 31 00:18:21.501482 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:21.501455    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod165b1a7a_d8ce_49d4_9a08_95a3cbea4f49.slice/crio-e0b3b13ccfd639cd31f61fb38d6c5c9d4434b54ca45e016d58b8929ac0384cc0 WatchSource:0}: Error finding container e0b3b13ccfd639cd31f61fb38d6c5c9d4434b54ca45e016d58b8929ac0384cc0: Status 404 returned error can't find the container with id e0b3b13ccfd639cd31f61fb38d6c5c9d4434b54ca45e016d58b8929ac0384cc0
Oct 31 00:18:21.504167 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.504075    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g24ff"
Oct 31 00:18:21.509026 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.509002    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:18:21.511163 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.511144    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:18:21.516573 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:21.516543    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17401e80_fe41_417f_9840_acd7a68d17e3.slice/crio-c512452c6fa7183057ed9b9881a52da2709559b7b82122786b548786479c4c29 WatchSource:0}: Error finding container c512452c6fa7183057ed9b9881a52da2709559b7b82122786b548786479c4c29: Status 404 returned error can't find the container with id c512452c6fa7183057ed9b9881a52da2709559b7b82122786b548786479c4c29
Oct 31 00:18:21.520065 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:21.520039    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8190e88_1887_415f_95cb_d421f3fbc08b.slice/crio-dfa1b91dce127f457caac2591af59bec0d2ea3b18c56ff5080cabe0045f3e3ba WatchSource:0}: Error finding container dfa1b91dce127f457caac2591af59bec0d2ea3b18c56ff5080cabe0045f3e3ba: Status 404 returned error can't find the container with id dfa1b91dce127f457caac2591af59bec0d2ea3b18c56ff5080cabe0045f3e3ba
Oct 31 00:18:21.524393 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:21.524337    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f456866_ce05_42c8_adb6_588cf5fcd8b7.slice/crio-a7e2562eeb99722d42965488cc842515544f34ffd2d63363c1ede3bc8233d04b WatchSource:0}: Error finding container a7e2562eeb99722d42965488cc842515544f34ffd2d63363c1ede3bc8233d04b: Status 404 returned error can't find the container with id a7e2562eeb99722d42965488cc842515544f34ffd2d63363c1ede3bc8233d04b
Oct 31 00:18:21.841823 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.841788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/002d2da2-bb18-4667-98c8-1770b70d9985-original-pull-secret\") pod \"global-pull-secret-syncer-pncsr\" (UID: \"002d2da2-bb18-4667-98c8-1770b70d9985\") " pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:21.842002 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.841854    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-metrics-certs\") pod \"network-metrics-daemon-5tw4x\" (UID: \"3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a\") " pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:21.842002 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:21.841962    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:18:21.842002 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:21.841975    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:18:21.842153 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:21.842036    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-metrics-certs podName:3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:22.842009315 +0000 UTC m=+4.154408578 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-metrics-certs") pod "network-metrics-daemon-5tw4x" (UID: "3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:18:21.842153 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:21.842050    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/002d2da2-bb18-4667-98c8-1770b70d9985-original-pull-secret podName:002d2da2-bb18-4667-98c8-1770b70d9985 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:22.842043877 +0000 UTC m=+4.154443127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/002d2da2-bb18-4667-98c8-1770b70d9985-original-pull-secret") pod "global-pull-secret-syncer-pncsr" (UID: "002d2da2-bb18-4667-98c8-1770b70d9985") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:18:21.942942 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:21.942641    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngkbc\" (UniqueName: \"kubernetes.io/projected/5b9f4a03-68eb-418e-9447-ad62473a1e9a-kube-api-access-ngkbc\") pod \"network-check-target-vdxch\" (UID: \"5b9f4a03-68eb-418e-9447-ad62473a1e9a\") " pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:21.942942 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:21.942934    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:18:21.943151 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:21.942949    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:18:21.943151 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:21.942959    2580 projected.go:196] Error preparing data for projected volume kube-api-access-ngkbc for pod openshift-network-diagnostics/network-check-target-vdxch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:18:21.943151 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:21.943003    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b9f4a03-68eb-418e-9447-ad62473a1e9a-kube-api-access-ngkbc podName:5b9f4a03-68eb-418e-9447-ad62473a1e9a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:22.942990777 +0000 UTC m=+4.255390031 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngkbc" (UniqueName: "kubernetes.io/projected/5b9f4a03-68eb-418e-9447-ad62473a1e9a-kube-api-access-ngkbc") pod "network-check-target-vdxch" (UID: "5b9f4a03-68eb-418e-9447-ad62473a1e9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:18:22.182483 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:22.182350    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:13:21 +0000 UTC" deadline="2028-07-18 10:29:51.625823176 +0000 UTC"
Oct 31 00:18:22.182483 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:22.182418    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15034h11m29.443409428s"
Oct 31 00:18:22.212649 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:22.211858    2580 generic.go:356] "Generic (PLEG): container finished" podID="c35a4f872afa19338c317775c3e775c3" containerID="8b59e0cb2290f46f406233a8a8b14a37f3bba28af7d00c23a39b289387ccfa13" exitCode=0
Oct 31 00:18:22.212649 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:22.211956    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal" event={"ID":"c35a4f872afa19338c317775c3e775c3","Type":"ContainerDied","Data":"8b59e0cb2290f46f406233a8a8b14a37f3bba28af7d00c23a39b289387ccfa13"}
Oct 31 00:18:22.224749 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:22.224720    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal" event={"ID":"dd74f5dcc55b4e9100e01ca71d574990","Type":"ContainerStarted","Data":"2857a4cdce295468836e3b02642f847bb3932773aac780051e7c79c8d1b79a7c"}
Oct 31 00:18:22.245991 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:22.245958    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qk5kf" event={"ID":"1f456866-ce05-42c8-adb6-588cf5fcd8b7","Type":"ContainerStarted","Data":"a7e2562eeb99722d42965488cc842515544f34ffd2d63363c1ede3bc8233d04b"}
Oct 31 00:18:22.255698 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:22.254402    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9d69r" event={"ID":"ec6a6238-81be-4a82-b346-b4c9fe3aefa4","Type":"ContainerStarted","Data":"c20b6ee53b5607496b962c020108ac6e9c8847d050b4c924742840c01fb84bed"}
Oct 31 00:18:22.265288 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:22.265156    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7" event={"ID":"b0c02c18-5891-493e-ab22-5191677b8364","Type":"ContainerStarted","Data":"8c8dc8b6dcc79627f55cf1e5bff9ad8dc3e3ed9eb7e1cf8f4ad9d88aecd707eb"}
Oct 31 00:18:22.269243 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:22.269163    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pwt7t" event={"ID":"a8190e88-1887-415f-95cb-d421f3fbc08b","Type":"ContainerStarted","Data":"dfa1b91dce127f457caac2591af59bec0d2ea3b18c56ff5080cabe0045f3e3ba"}
Oct 31 00:18:22.285528 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:22.285393    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g24ff" event={"ID":"17401e80-fe41-417f-9840-acd7a68d17e3","Type":"ContainerStarted","Data":"c512452c6fa7183057ed9b9881a52da2709559b7b82122786b548786479c4c29"}
Oct 31 00:18:22.290084 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:22.289995    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g" event={"ID":"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49","Type":"ContainerStarted","Data":"e0b3b13ccfd639cd31f61fb38d6c5c9d4434b54ca45e016d58b8929ac0384cc0"}
Oct 31 00:18:22.294295 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:22.294162    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal" podStartSLOduration=2.29414841 podStartE2EDuration="2.29414841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:22.293999675 +0000 UTC m=+3.606398948" watchObservedRunningTime="2026-10-31 00:18:22.29414841 +0000 UTC m=+3.606547681"
Oct 31 00:18:22.297829 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:22.297745    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6ljg5" event={"ID":"bbc0fbb5-0386-40b9-aeef-3b7aa867f625","Type":"ContainerStarted","Data":"b73bdbdfbf94a552b042bbcfb5d528e3de51d3cf0c5f5b7ec764f58971cdc684"}
Oct 31 00:18:22.329152 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:22.329103    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-htbtd" event={"ID":"95f3be30-2d65-4373-9623-155c841dca11","Type":"ContainerStarted","Data":"11ce4c9c645862b529ca4e56f87a3c0335b7172e5515b8ab3b93d995e3942425"}
Oct 31 00:18:22.332032 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:22.331989    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9j7z8" event={"ID":"4fdd0807-bb6c-4867-aad0-40af2ddc7d08","Type":"ContainerStarted","Data":"cf16bf06fb01a647c3bfe579684dc0d24ef6a9de7c246b2befb30a90bfd3fe58"}
Oct 31 00:18:22.851961 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:22.851158    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-metrics-certs\") pod \"network-metrics-daemon-5tw4x\" (UID: \"3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a\") " pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:22.851961 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:22.851226    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/002d2da2-bb18-4667-98c8-1770b70d9985-original-pull-secret\") pod \"global-pull-secret-syncer-pncsr\" (UID: \"002d2da2-bb18-4667-98c8-1770b70d9985\") " pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:22.851961 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:22.851373    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:18:22.851961 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:22.851434    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/002d2da2-bb18-4667-98c8-1770b70d9985-original-pull-secret podName:002d2da2-bb18-4667-98c8-1770b70d9985 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:24.851416206 +0000 UTC m=+6.163815459 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/002d2da2-bb18-4667-98c8-1770b70d9985-original-pull-secret") pod "global-pull-secret-syncer-pncsr" (UID: "002d2da2-bb18-4667-98c8-1770b70d9985") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:18:22.851961 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:22.851873    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:18:22.851961 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:22.851925    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-metrics-certs podName:3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:24.851909184 +0000 UTC m=+6.164308484 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-metrics-certs") pod "network-metrics-daemon-5tw4x" (UID: "3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:18:22.952332 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:22.951752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngkbc\" (UniqueName: \"kubernetes.io/projected/5b9f4a03-68eb-418e-9447-ad62473a1e9a-kube-api-access-ngkbc\") pod \"network-check-target-vdxch\" (UID: \"5b9f4a03-68eb-418e-9447-ad62473a1e9a\") " pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:22.952332 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:22.951926    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:18:22.952332 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:22.951944    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:18:22.952332 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:22.951957    2580 projected.go:196] Error preparing data for projected volume kube-api-access-ngkbc for pod openshift-network-diagnostics/network-check-target-vdxch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:18:22.952332 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:22.952015    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b9f4a03-68eb-418e-9447-ad62473a1e9a-kube-api-access-ngkbc podName:5b9f4a03-68eb-418e-9447-ad62473a1e9a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:24.951998899 +0000 UTC m=+6.264398162 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngkbc" (UniqueName: "kubernetes.io/projected/5b9f4a03-68eb-418e-9447-ad62473a1e9a-kube-api-access-ngkbc") pod "network-check-target-vdxch" (UID: "5b9f4a03-68eb-418e-9447-ad62473a1e9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:18:23.183314 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:23.183224    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:13:21 +0000 UTC" deadline="2028-04-12 02:09:07.831203419 +0000 UTC"
Oct 31 00:18:23.183314 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:23.183267    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12697h50m44.647941459s"
Oct 31 00:18:23.199495 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:23.199468    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:23.199637 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:23.199610    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5tw4x" podUID="3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a"
Oct 31 00:18:23.200053 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:23.200031    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:23.200159 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:23.200137    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdxch" podUID="5b9f4a03-68eb-418e-9447-ad62473a1e9a"
Oct 31 00:18:23.200326 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:23.200299    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:23.200425 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:23.200406    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pncsr" podUID="002d2da2-bb18-4667-98c8-1770b70d9985"
Oct 31 00:18:23.349520 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:23.349479    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal" event={"ID":"c35a4f872afa19338c317775c3e775c3","Type":"ContainerStarted","Data":"e8bedbeece192ade68729ec36f0da3240f18edf9a66e29ddf3401d8229657d79"}
Oct 31 00:18:24.364341 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:24.362946    2580 generic.go:356] "Generic (PLEG): container finished" podID="a8190e88-1887-415f-95cb-d421f3fbc08b" containerID="02b1aa7b5186e1f998028e17a26cef03929486e1d654f0a2b2ca0bd4bb242815" exitCode=0
Oct 31 00:18:24.364341 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:24.363995    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pwt7t" event={"ID":"a8190e88-1887-415f-95cb-d421f3fbc08b","Type":"ContainerDied","Data":"02b1aa7b5186e1f998028e17a26cef03929486e1d654f0a2b2ca0bd4bb242815"}
Oct 31 00:18:24.439292 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:24.439211    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal" podStartSLOduration=4.439192422 podStartE2EDuration="4.439192422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:23.391480056 +0000 UTC m=+4.703879330" watchObservedRunningTime="2026-10-31 00:18:24.439192422 +0000 UTC m=+5.751591698"
Oct 31 00:18:24.870606 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:24.870572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/002d2da2-bb18-4667-98c8-1770b70d9985-original-pull-secret\") pod \"global-pull-secret-syncer-pncsr\" (UID: \"002d2da2-bb18-4667-98c8-1770b70d9985\") " pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:24.870802 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:24.870634    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-metrics-certs\") pod \"network-metrics-daemon-5tw4x\" (UID: \"3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a\") " pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:24.870802 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:24.870776    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:18:24.870928 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:24.870840    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-metrics-certs podName:3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:28.870821399 +0000 UTC m=+10.183220650 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-metrics-certs") pod "network-metrics-daemon-5tw4x" (UID: "3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:18:24.871431 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:24.871279    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:18:24.871431 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:24.871332    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/002d2da2-bb18-4667-98c8-1770b70d9985-original-pull-secret podName:002d2da2-bb18-4667-98c8-1770b70d9985 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:28.87131751 +0000 UTC m=+10.183716759 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/002d2da2-bb18-4667-98c8-1770b70d9985-original-pull-secret") pod "global-pull-secret-syncer-pncsr" (UID: "002d2da2-bb18-4667-98c8-1770b70d9985") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:18:24.971662 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:24.971619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngkbc\" (UniqueName: \"kubernetes.io/projected/5b9f4a03-68eb-418e-9447-ad62473a1e9a-kube-api-access-ngkbc\") pod \"network-check-target-vdxch\" (UID: \"5b9f4a03-68eb-418e-9447-ad62473a1e9a\") " pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:24.971867 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:24.971858    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:18:24.971932 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:24.971873    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:18:24.971932 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:24.971884    2580 projected.go:196] Error preparing data for projected volume kube-api-access-ngkbc for pod openshift-network-diagnostics/network-check-target-vdxch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:18:24.971932 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:24.971930    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b9f4a03-68eb-418e-9447-ad62473a1e9a-kube-api-access-ngkbc podName:5b9f4a03-68eb-418e-9447-ad62473a1e9a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:28.971917627 +0000 UTC m=+10.284316877 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngkbc" (UniqueName: "kubernetes.io/projected/5b9f4a03-68eb-418e-9447-ad62473a1e9a-kube-api-access-ngkbc") pod "network-check-target-vdxch" (UID: "5b9f4a03-68eb-418e-9447-ad62473a1e9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:18:25.198102 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:25.198021    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:25.198271 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:25.198164    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5tw4x" podUID="3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a"
Oct 31 00:18:25.200915 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:25.200852    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:25.201051 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:25.200960    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pncsr" podUID="002d2da2-bb18-4667-98c8-1770b70d9985"
Oct 31 00:18:25.201051 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:25.200989    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:25.201152 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:25.201050    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdxch" podUID="5b9f4a03-68eb-418e-9447-ad62473a1e9a"
Oct 31 00:18:26.373689 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:26.373614    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g24ff" event={"ID":"17401e80-fe41-417f-9840-acd7a68d17e3","Type":"ContainerStarted","Data":"93589f2b4dec25e7758bf726946b82d91a59a9b9b18c5c6b5ec135e21c751f4c"}
Oct 31 00:18:27.198566 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:27.198047    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:27.198566 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:27.198196    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5tw4x" podUID="3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a"
Oct 31 00:18:27.198566 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:27.198469    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:27.198566 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:27.198532    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdxch" podUID="5b9f4a03-68eb-418e-9447-ad62473a1e9a"
Oct 31 00:18:27.199318 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:27.199133    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:27.199318 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:27.199234    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pncsr" podUID="002d2da2-bb18-4667-98c8-1770b70d9985"
Oct 31 00:18:27.378441 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:27.378358    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g" event={"ID":"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49","Type":"ContainerStarted","Data":"f19fbccda1522c295afbfb6722649a8dc61549d37094e7f653eccbbee0755ade"}
Oct 31 00:18:27.383093 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:27.383063    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-htbtd" event={"ID":"95f3be30-2d65-4373-9623-155c841dca11","Type":"ContainerStarted","Data":"7aca195989d6619b406bbe7a56589848f8e83d84f150e497d96da851069e9b2f"}
Oct 31 00:18:27.424199 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:27.424119    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g24ff" podStartSLOduration=4.633551976 podStartE2EDuration="8.424100057s" totalImagesPullingTime="3.790548081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:26.415284322 +0000 UTC m=+7.727683595" watchObservedRunningTime="2026-10-31 00:18:27.424100057 +0000 UTC m=+8.736499330"
Oct 31 00:18:28.591181 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:28.591162    2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 00:18:28.913224 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:28.913190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-metrics-certs\") pod \"network-metrics-daemon-5tw4x\" (UID: \"3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a\") " pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:28.913337 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:28.913241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/002d2da2-bb18-4667-98c8-1770b70d9985-original-pull-secret\") pod \"global-pull-secret-syncer-pncsr\" (UID: \"002d2da2-bb18-4667-98c8-1770b70d9985\") " pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:28.913405 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:28.913387    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:18:28.913470 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:28.913455    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:18:28.913470 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:28.913467    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-metrics-certs podName:3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:36.913446107 +0000 UTC m=+18.225845373 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-metrics-certs") pod "network-metrics-daemon-5tw4x" (UID: "3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:18:28.913576 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:28.913524    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/002d2da2-bb18-4667-98c8-1770b70d9985-original-pull-secret podName:002d2da2-bb18-4667-98c8-1770b70d9985 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:36.913507427 +0000 UTC m=+18.225906680 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/002d2da2-bb18-4667-98c8-1770b70d9985-original-pull-secret") pod "global-pull-secret-syncer-pncsr" (UID: "002d2da2-bb18-4667-98c8-1770b70d9985") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:18:29.014605 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:29.014565    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngkbc\" (UniqueName: \"kubernetes.io/projected/5b9f4a03-68eb-418e-9447-ad62473a1e9a-kube-api-access-ngkbc\") pod \"network-check-target-vdxch\" (UID: \"5b9f4a03-68eb-418e-9447-ad62473a1e9a\") " pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:29.014841 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:29.014825    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:18:29.014915 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:29.014851    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:18:29.014915 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:29.014864    2580 projected.go:196] Error preparing data for projected volume kube-api-access-ngkbc for pod openshift-network-diagnostics/network-check-target-vdxch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:18:29.015015 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:29.014922    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b9f4a03-68eb-418e-9447-ad62473a1e9a-kube-api-access-ngkbc podName:5b9f4a03-68eb-418e-9447-ad62473a1e9a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:37.014904207 +0000 UTC m=+18.327303463 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngkbc" (UniqueName: "kubernetes.io/projected/5b9f4a03-68eb-418e-9447-ad62473a1e9a-kube-api-access-ngkbc") pod "network-check-target-vdxch" (UID: "5b9f4a03-68eb-418e-9447-ad62473a1e9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:18:29.187262 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:29.187184    2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:18:28.591177743Z","UUID":"9bb87411-fd97-4ab6-9d88-378de19d596d","Handler":null,"Name":"","Endpoint":""}
Oct 31 00:18:29.191845 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:29.189344    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 00:18:29.191845 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:29.189383    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 00:18:29.201008 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:29.200986    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:29.201131 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:29.201080    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdxch" podUID="5b9f4a03-68eb-418e-9447-ad62473a1e9a"
Oct 31 00:18:29.201207 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:29.201145    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:29.201259 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:29.201227    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5tw4x" podUID="3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a"
Oct 31 00:18:29.201319 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:29.201276    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:29.201367 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:29.201347    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pncsr" podUID="002d2da2-bb18-4667-98c8-1770b70d9985"
Oct 31 00:18:29.388928 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:29.388890    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6ljg5" event={"ID":"bbc0fbb5-0386-40b9-aeef-3b7aa867f625","Type":"ContainerStarted","Data":"961ac7c342f5f5120d9766b3bf482defbc42777128e4e0b2e42f586a012973a2"}
Oct 31 00:18:29.392201 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:29.392174    2580 generic.go:356] "Generic (PLEG): container finished" podID="a8190e88-1887-415f-95cb-d421f3fbc08b" containerID="af80054aab992f1e709d5820d15617f7426022967f4b9d39b820e92aff281c7f" exitCode=0
Oct 31 00:18:29.392315 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:29.392239    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pwt7t" event={"ID":"a8190e88-1887-415f-95cb-d421f3fbc08b","Type":"ContainerDied","Data":"af80054aab992f1e709d5820d15617f7426022967f4b9d39b820e92aff281c7f"}
Oct 31 00:18:29.394436 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:29.394417    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g" event={"ID":"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49","Type":"ContainerStarted","Data":"0619b9757bc307723125626288793badfd2f3e4571880dfd883e6e9b4d9af0a8"}
Oct 31 00:18:29.433171 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:29.433075    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-6ljg5" podStartSLOduration=3.339554532 podStartE2EDuration="10.433060828s" totalImagesPullingTime="7.093506296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:29.433043287 +0000 UTC m=+10.745442564" watchObservedRunningTime="2026-10-31 00:18:29.433060828 +0000 UTC m=+10.745460099"
Oct 31 00:18:29.433428 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:29.433400    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-htbtd" podStartSLOduration=5.721994176 podStartE2EDuration="10.433394722s" totalImagesPullingTime="4.711400546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:27.424512071 +0000 UTC m=+8.736911344" watchObservedRunningTime="2026-10-31 00:18:29.433394722 +0000 UTC m=+10.745793993"
Oct 31 00:18:29.728231 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:29.728191    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g24ff"
Oct 31 00:18:29.728890 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:29.728871    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g24ff"
Oct 31 00:18:30.399046 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:30.398905    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9d69r" event={"ID":"ec6a6238-81be-4a82-b346-b4c9fe3aefa4","Type":"ContainerStarted","Data":"0583f0d01fcd53b61c793bde3308f83bb983ec8bb315ca43aa21e9113c1f2eaf"}
Oct 31 00:18:30.401485 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:30.401453    2580 generic.go:356] "Generic (PLEG): container finished" podID="a8190e88-1887-415f-95cb-d421f3fbc08b" containerID="0981b2fe61bcf2135ad2bc877c45a99fabeadb8d62090596889d47d50025f45c" exitCode=0
Oct 31 00:18:30.401608 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:30.401537    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pwt7t" event={"ID":"a8190e88-1887-415f-95cb-d421f3fbc08b","Type":"ContainerDied","Data":"0981b2fe61bcf2135ad2bc877c45a99fabeadb8d62090596889d47d50025f45c"}
Oct 31 00:18:30.405480 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:30.405453    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g" event={"ID":"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49","Type":"ContainerStarted","Data":"ed2784f0fe590cf90b63d33822c4b94db74e4d28a702f15156e67a2696c4fd8f"}
Oct 31 00:18:30.440501 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:30.440437    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-9d69r" podStartSLOduration=3.4840453350000002 podStartE2EDuration="11.44042072s" totalImagesPullingTime="7.956375385s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:30.440240717 +0000 UTC m=+11.752639989" watchObservedRunningTime="2026-10-31 00:18:30.44042072 +0000 UTC m=+11.752819991"
Oct 31 00:18:30.532946 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:30.532894    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g" podStartSLOduration=3.786095623 podStartE2EDuration="11.532880585s" totalImagesPullingTime="7.746784962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:30.483132843 +0000 UTC m=+11.795532115" watchObservedRunningTime="2026-10-31 00:18:30.532880585 +0000 UTC m=+11.845279858"
Oct 31 00:18:31.198673 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:31.198641    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:31.199162 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:31.198641    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:31.199162 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:31.198762    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:31.199162 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:31.198766    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pncsr" podUID="002d2da2-bb18-4667-98c8-1770b70d9985"
Oct 31 00:18:31.199162 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:31.198832    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdxch" podUID="5b9f4a03-68eb-418e-9447-ad62473a1e9a"
Oct 31 00:18:31.199162 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:31.198937    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5tw4x" podUID="3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a"
Oct 31 00:18:31.409428 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:31.409337    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9j7z8" event={"ID":"4fdd0807-bb6c-4867-aad0-40af2ddc7d08","Type":"ContainerStarted","Data":"1d349eed5bce326f77695b0a473a4676482849b1e356dfc1926ab89c91ac173d"}
Oct 31 00:18:31.412148 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:31.412020    2580 generic.go:356] "Generic (PLEG): container finished" podID="a8190e88-1887-415f-95cb-d421f3fbc08b" containerID="547fccdc6942c91f27791a17df1847c3298813d349c1b82b9622c7803728be0c" exitCode=0
Oct 31 00:18:31.412148 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:31.412083    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pwt7t" event={"ID":"a8190e88-1887-415f-95cb-d421f3fbc08b","Type":"ContainerDied","Data":"547fccdc6942c91f27791a17df1847c3298813d349c1b82b9622c7803728be0c"}
Oct 31 00:18:31.451944 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:31.450776    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9j7z8" podStartSLOduration=5.019871492 podStartE2EDuration="12.450757743s" totalImagesPullingTime="7.430886251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:31.450577896 +0000 UTC m=+12.762977162" watchObservedRunningTime="2026-10-31 00:18:31.450757743 +0000 UTC m=+12.763157017"
Oct 31 00:18:32.302558 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:32.302518    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g24ff"
Oct 31 00:18:32.303167 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:32.302727    2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 00:18:32.303167 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:32.303157    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g24ff"
Oct 31 00:18:32.417041 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:32.417002    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qk5kf" event={"ID":"1f456866-ce05-42c8-adb6-588cf5fcd8b7","Type":"ContainerStarted","Data":"360a36878c73bc15fc0ad393be2bbfe5a1e2512a9a6c062350ae7631e951c51b"}
Oct 31 00:18:33.198926 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:33.198655    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:33.198926 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:33.198687    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:33.199129 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:33.198931    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pncsr" podUID="002d2da2-bb18-4667-98c8-1770b70d9985"
Oct 31 00:18:33.199129 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:33.198699    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:33.199129 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:33.199036    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5tw4x" podUID="3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a"
Oct 31 00:18:33.199129 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:33.199109    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdxch" podUID="5b9f4a03-68eb-418e-9447-ad62473a1e9a"
Oct 31 00:18:33.422011 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:33.421965    2580 generic.go:356] "Generic (PLEG): container finished" podID="a8190e88-1887-415f-95cb-d421f3fbc08b" containerID="bcd17679cef98b9db708087c03cb9af8e7d54f0db7a099e92bc917f106262185" exitCode=0
Oct 31 00:18:33.422575 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:33.422056    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pwt7t" event={"ID":"a8190e88-1887-415f-95cb-d421f3fbc08b","Type":"ContainerDied","Data":"bcd17679cef98b9db708087c03cb9af8e7d54f0db7a099e92bc917f106262185"}
Oct 31 00:18:33.509047 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:33.508587    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qk5kf" podStartSLOduration=4.60303515 podStartE2EDuration="14.50856739s" totalImagesPullingTime="9.90553224s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:32.464930065 +0000 UTC m=+13.777329338" watchObservedRunningTime="2026-10-31 00:18:33.50856739 +0000 UTC m=+14.820966663"
Oct 31 00:18:34.428272 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:34.428233    2580 generic.go:356] "Generic (PLEG): container finished" podID="a8190e88-1887-415f-95cb-d421f3fbc08b" containerID="fa650fa93fd7589469468396212d259afe700a261c7abbc683602ae82170cf20" exitCode=0
Oct 31 00:18:34.428803 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:34.428318    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pwt7t" event={"ID":"a8190e88-1887-415f-95cb-d421f3fbc08b","Type":"ContainerDied","Data":"fa650fa93fd7589469468396212d259afe700a261c7abbc683602ae82170cf20"}
Oct 31 00:18:35.197719 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:35.197674    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:35.197719 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:35.197702    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:35.197933 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:35.197834    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdxch" podUID="5b9f4a03-68eb-418e-9447-ad62473a1e9a"
Oct 31 00:18:35.197933 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:35.197894    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:35.198056 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:35.198027    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5tw4x" podUID="3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a"
Oct 31 00:18:35.198139 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:35.198118    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pncsr" podUID="002d2da2-bb18-4667-98c8-1770b70d9985"
Oct 31 00:18:35.433359 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:35.433326    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pwt7t" event={"ID":"a8190e88-1887-415f-95cb-d421f3fbc08b","Type":"ContainerStarted","Data":"5ba48a43479cdb63dea905c8264547888a8ee5cc3f5c9ab60c3914e1f8604328"}
Oct 31 00:18:35.507622 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:35.507522    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pwt7t" podStartSLOduration=6.847972342 podStartE2EDuration="16.507509503s" totalImagesPullingTime="9.659537161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:35.507227724 +0000 UTC m=+16.819626996" watchObservedRunningTime="2026-10-31 00:18:35.507509503 +0000 UTC m=+16.819908775"
Oct 31 00:18:36.438828 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:36.438569    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7" event={"ID":"b0c02c18-5891-493e-ab22-5191677b8364","Type":"ContainerStarted","Data":"4e6eb1024cab4ae567619b00b09ec42e812d5c55d9bd8d48c1156c49b4c07b08"}
Oct 31 00:18:36.439281 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:36.438847    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7" event={"ID":"b0c02c18-5891-493e-ab22-5191677b8364","Type":"ContainerStarted","Data":"560aa3475beb859a62c2c13b9773c533275e1198975d6c1e8e060fb9930173b4"}
Oct 31 00:18:36.439281 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:36.438888    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7" event={"ID":"b0c02c18-5891-493e-ab22-5191677b8364","Type":"ContainerStarted","Data":"4df9cf7afabea0ed1c70094fbc168fe4c5dab0d60ff995084a3fbbcc0b5dc942"}
Oct 31 00:18:36.439281 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:36.438916    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7" event={"ID":"b0c02c18-5891-493e-ab22-5191677b8364","Type":"ContainerStarted","Data":"e73f0e5e6a66c37900ae6e058207327368b3adb038ce5f186c1cedf33b0abfa2"}
Oct 31 00:18:36.439281 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:36.438944    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7" event={"ID":"b0c02c18-5891-493e-ab22-5191677b8364","Type":"ContainerStarted","Data":"5196fd1de7154c3911a1da420085469945d3a0eccea439eea0743beebadfad03"}
Oct 31 00:18:36.439281 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:36.438974    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7" event={"ID":"b0c02c18-5891-493e-ab22-5191677b8364","Type":"ContainerStarted","Data":"3513708a08e9fa85864e07bc23837cc3d9947bd312de3c7841532d946174ee33"}
Oct 31 00:18:36.975611 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:36.975572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-metrics-certs\") pod \"network-metrics-daemon-5tw4x\" (UID: \"3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a\") " pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:36.975801 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:36.975622    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/002d2da2-bb18-4667-98c8-1770b70d9985-original-pull-secret\") pod \"global-pull-secret-syncer-pncsr\" (UID: \"002d2da2-bb18-4667-98c8-1770b70d9985\") " pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:36.975801 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:36.975744    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:18:36.975881 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:36.975802    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/002d2da2-bb18-4667-98c8-1770b70d9985-original-pull-secret podName:002d2da2-bb18-4667-98c8-1770b70d9985 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:52.975787706 +0000 UTC m=+34.288186956 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/002d2da2-bb18-4667-98c8-1770b70d9985-original-pull-secret") pod "global-pull-secret-syncer-pncsr" (UID: "002d2da2-bb18-4667-98c8-1770b70d9985") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:18:36.975881 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:36.975744    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:18:36.975881 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:36.975873    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-metrics-certs podName:3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:52.975861826 +0000 UTC m=+34.288261084 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-metrics-certs") pod "network-metrics-daemon-5tw4x" (UID: "3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:18:37.076579 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:37.076539    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngkbc\" (UniqueName: \"kubernetes.io/projected/5b9f4a03-68eb-418e-9447-ad62473a1e9a-kube-api-access-ngkbc\") pod \"network-check-target-vdxch\" (UID: \"5b9f4a03-68eb-418e-9447-ad62473a1e9a\") " pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:37.076766 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:37.076742    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:18:37.076766 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:37.076765    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:18:37.076833 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:37.076776    2580 projected.go:196] Error preparing data for projected volume kube-api-access-ngkbc for pod openshift-network-diagnostics/network-check-target-vdxch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:18:37.076865 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:37.076833    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b9f4a03-68eb-418e-9447-ad62473a1e9a-kube-api-access-ngkbc podName:5b9f4a03-68eb-418e-9447-ad62473a1e9a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:53.076819553 +0000 UTC m=+34.389218806 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngkbc" (UniqueName: "kubernetes.io/projected/5b9f4a03-68eb-418e-9447-ad62473a1e9a-kube-api-access-ngkbc") pod "network-check-target-vdxch" (UID: "5b9f4a03-68eb-418e-9447-ad62473a1e9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:18:37.197780 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:37.197743    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:37.197946 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:37.197743    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:37.197946 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:37.197859    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pncsr" podUID="002d2da2-bb18-4667-98c8-1770b70d9985"
Oct 31 00:18:37.197946 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:37.197909    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdxch" podUID="5b9f4a03-68eb-418e-9447-ad62473a1e9a"
Oct 31 00:18:37.197946 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:37.197759    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:37.198077 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:37.198052    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5tw4x" podUID="3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a"
Oct 31 00:18:38.445433 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:38.445393    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7" event={"ID":"b0c02c18-5891-493e-ab22-5191677b8364","Type":"ContainerStarted","Data":"c284f10b28c2a3a907f55232aa0b351e83aa01fa8ef9be56e177bddd7b282416"}
Oct 31 00:18:39.199941 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:39.199899    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:39.200115 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:39.199948    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:39.200115 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:39.200030    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:39.200115 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:39.200044    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5tw4x" podUID="3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a"
Oct 31 00:18:39.200216 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:39.200110    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pncsr" podUID="002d2da2-bb18-4667-98c8-1770b70d9985"
Oct 31 00:18:39.200216 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:39.200197    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdxch" podUID="5b9f4a03-68eb-418e-9447-ad62473a1e9a"
Oct 31 00:18:40.454048 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:40.454007    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7" event={"ID":"b0c02c18-5891-493e-ab22-5191677b8364","Type":"ContainerStarted","Data":"41a27d34d62752f8aafc6d8e6743801da36ff1969d8caa527345e9f3679ff145"}
Oct 31 00:18:40.454777 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:40.454398    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:40.454777 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:40.454476    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:40.471197 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:40.470962    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:40.471197 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:40.471127    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:40.554158 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:40.553963    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7" podStartSLOduration=7.126916051 podStartE2EDuration="21.553949569s" totalImagesPullingTime="14.427033518s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:40.553564497 +0000 UTC m=+21.865963769" watchObservedRunningTime="2026-10-31 00:18:40.553949569 +0000 UTC m=+21.866348858"
Oct 31 00:18:41.198152 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:41.198122    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:41.198152 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:41.198145    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:41.198345 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:41.198231    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pncsr" podUID="002d2da2-bb18-4667-98c8-1770b70d9985"
Oct 31 00:18:41.198345 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:41.198273    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:41.198410 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:41.198344    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdxch" podUID="5b9f4a03-68eb-418e-9447-ad62473a1e9a"
Oct 31 00:18:41.198447 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:41.198417    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5tw4x" podUID="3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a"
Oct 31 00:18:41.456759 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:41.456664    2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 00:18:42.459372 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:42.459300    2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 00:18:42.520855 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:42.520821    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pncsr"]
Oct 31 00:18:42.521023 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:42.520983    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:42.521117 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:42.521091    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pncsr" podUID="002d2da2-bb18-4667-98c8-1770b70d9985"
Oct 31 00:18:42.526932 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:42.526906    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vdxch"]
Oct 31 00:18:42.527100 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:42.527032    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:42.527161 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:42.527134    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdxch" podUID="5b9f4a03-68eb-418e-9447-ad62473a1e9a"
Oct 31 00:18:42.534392 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:42.534365    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5tw4x"]
Oct 31 00:18:42.534522 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:42.534488    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:42.534596 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:42.534579    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5tw4x" podUID="3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a"
Oct 31 00:18:44.198296 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:44.198263    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:44.198790 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:44.198276    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:44.198790 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:44.198368    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pncsr" podUID="002d2da2-bb18-4667-98c8-1770b70d9985"
Oct 31 00:18:44.198790 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:44.198449    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdxch" podUID="5b9f4a03-68eb-418e-9447-ad62473a1e9a"
Oct 31 00:18:44.198790 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:44.198276    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:44.198790 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:44.198528    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5tw4x" podUID="3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a"
Oct 31 00:18:46.197807 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:46.197776    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:46.198176 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:46.197776    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:46.198176 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:46.197885    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdxch" podUID="5b9f4a03-68eb-418e-9447-ad62473a1e9a"
Oct 31 00:18:46.198176 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:46.197776    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:46.198176 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:46.197956    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pncsr" podUID="002d2da2-bb18-4667-98c8-1770b70d9985"
Oct 31 00:18:46.198176 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:46.198020    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5tw4x" podUID="3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a"
Oct 31 00:18:48.198666 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.198633    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:48.199037 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.198633    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:48.199037 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.198746    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:48.199037 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:48.198756    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5tw4x" podUID="3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a"
Oct 31 00:18:48.199037 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:48.198830    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdxch" podUID="5b9f4a03-68eb-418e-9447-ad62473a1e9a"
Oct 31 00:18:48.199037 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:48.198918    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pncsr" podUID="002d2da2-bb18-4667-98c8-1770b70d9985"
Oct 31 00:18:48.494221 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.494193    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeReady"
Oct 31 00:18:48.494381 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.494341    2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 00:18:48.566866 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.566838    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vv2qm"]
Oct 31 00:18:48.624443 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.624413    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-h26l7"]
Oct 31 00:18:48.624627 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.624549    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:18:48.642518 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.642485    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vv2qm"]
Oct 31 00:18:48.642645 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.642533    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-v8ft8"]
Oct 31 00:18:48.642645 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.642593    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h26l7"
Oct 31 00:18:48.646424 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.646399    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 00:18:48.646537 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.646454    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5c7xv\""
Oct 31 00:18:48.651058 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.651041    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 00:18:48.652451 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.652433    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:18:48.656443 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.656423    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 00:18:48.656876 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.656859    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 00:18:48.658134 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.658115    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h26l7"]
Oct 31 00:18:48.696291 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.696267    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v8ft8"]
Oct 31 00:18:48.713113 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.713086    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-lpntz"]
Oct 31 00:18:48.736485 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.736446    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-lpntz"
Oct 31 00:18:48.736780 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.736759    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-lpntz"]
Oct 31 00:18:48.746649 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.746598    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 00:18:48.746649 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.746599    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 00:18:48.746793 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.746771    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rltkk\""
Oct 31 00:18:48.746827 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.746819    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-l44c8\""
Oct 31 00:18:48.750668 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.750654    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 00:18:48.750748 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.750736    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 00:18:48.750779 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.750752    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 00:18:48.755120 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.755105    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 00:18:48.755181 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.755158    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 00:18:48.755181 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.755171    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 00:18:48.755181 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.755174    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 00:18:48.755272 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.755228    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-wc6r2\""
Oct 31 00:18:48.759440 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.759425    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79ab34ed-5163-4cff-8cf4-170c3445e655-tmp-dir\") pod \"dns-default-v8ft8\" (UID: \"79ab34ed-5163-4cff-8cf4-170c3445e655\") " pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:18:48.759488 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.759448    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvxxw\" (UniqueName: \"kubernetes.io/projected/79ab34ed-5163-4cff-8cf4-170c3445e655-kube-api-access-jvxxw\") pod \"dns-default-v8ft8\" (UID: \"79ab34ed-5163-4cff-8cf4-170c3445e655\") " pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:18:48.759527 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.759481    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84a754a9-1c06-4302-bb01-4f53a1e032d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:18:48.759560 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.759549    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84a754a9-1c06-4302-bb01-4f53a1e032d0-data-volume\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:18:48.759599 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.759582    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q65jt\" (UniqueName: \"kubernetes.io/projected/8072caf6-1730-4100-a87a-630ace22a370-kube-api-access-q65jt\") pod \"ingress-canary-h26l7\" (UID: \"8072caf6-1730-4100-a87a-630ace22a370\") " pod="openshift-ingress-canary/ingress-canary-h26l7"
Oct 31 00:18:48.759629 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.759600    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/79ab34ed-5163-4cff-8cf4-170c3445e655-metrics-tls\") pod \"dns-default-v8ft8\" (UID: \"79ab34ed-5163-4cff-8cf4-170c3445e655\") " pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:18:48.759720 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.759703    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8072caf6-1730-4100-a87a-630ace22a370-cert\") pod \"ingress-canary-h26l7\" (UID: \"8072caf6-1730-4100-a87a-630ace22a370\") " pod="openshift-ingress-canary/ingress-canary-h26l7"
Oct 31 00:18:48.759759 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.759732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84a754a9-1c06-4302-bb01-4f53a1e032d0-crio-socket\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:18:48.759759 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.759755    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwvkk\" (UniqueName: \"kubernetes.io/projected/84a754a9-1c06-4302-bb01-4f53a1e032d0-kube-api-access-fwvkk\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:18:48.759837 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.759799    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/79ab34ed-5163-4cff-8cf4-170c3445e655-config-volume\") pod \"dns-default-v8ft8\" (UID: \"79ab34ed-5163-4cff-8cf4-170c3445e655\") " pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:18:48.759837 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.759817    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84a754a9-1c06-4302-bb01-4f53a1e032d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:18:48.861016 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.860984    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8072caf6-1730-4100-a87a-630ace22a370-cert\") pod \"ingress-canary-h26l7\" (UID: \"8072caf6-1730-4100-a87a-630ace22a370\") " pod="openshift-ingress-canary/ingress-canary-h26l7"
Oct 31 00:18:48.861016 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.861022    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84a754a9-1c06-4302-bb01-4f53a1e032d0-crio-socket\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:18:48.861214 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.861045    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwvkk\" (UniqueName: \"kubernetes.io/projected/84a754a9-1c06-4302-bb01-4f53a1e032d0-kube-api-access-fwvkk\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:18:48.861214 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.861176    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/79ab34ed-5163-4cff-8cf4-170c3445e655-config-volume\") pod \"dns-default-v8ft8\" (UID: \"79ab34ed-5163-4cff-8cf4-170c3445e655\") " pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:18:48.861291 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.861211    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84a754a9-1c06-4302-bb01-4f53a1e032d0-crio-socket\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:18:48.861291 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.861222    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84a754a9-1c06-4302-bb01-4f53a1e032d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:18:48.861367 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.861288    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79ab34ed-5163-4cff-8cf4-170c3445e655-tmp-dir\") pod \"dns-default-v8ft8\" (UID: \"79ab34ed-5163-4cff-8cf4-170c3445e655\") " pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:18:48.861367 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:48.861311    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:18:48.861367 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.861314    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvxxw\" (UniqueName: \"kubernetes.io/projected/79ab34ed-5163-4cff-8cf4-170c3445e655-kube-api-access-jvxxw\") pod \"dns-default-v8ft8\" (UID: \"79ab34ed-5163-4cff-8cf4-170c3445e655\") " pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:18:48.861367 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.861338    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84a754a9-1c06-4302-bb01-4f53a1e032d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:18:48.861367 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:48.861365    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84a754a9-1c06-4302-bb01-4f53a1e032d0-insights-runtime-extractor-tls podName:84a754a9-1c06-4302-bb01-4f53a1e032d0 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:49.361347845 +0000 UTC m=+30.673747115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84a754a9-1c06-4302-bb01-4f53a1e032d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vv2qm" (UID: "84a754a9-1c06-4302-bb01-4f53a1e032d0") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:18:48.861587 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.861383    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f18d4e41-daf0-4be5-bcdb-c00d189419ee-signing-key\") pod \"service-ca-5fffc97bdf-lpntz\" (UID: \"f18d4e41-daf0-4be5-bcdb-c00d189419ee\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpntz"
Oct 31 00:18:48.861587 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:48.861395    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 00:18:48.861587 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.861424    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84a754a9-1c06-4302-bb01-4f53a1e032d0-data-volume\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:18:48.861587 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:48.861437    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84a754a9-1c06-4302-bb01-4f53a1e032d0-kube-rbac-proxy-cm podName:84a754a9-1c06-4302-bb01-4f53a1e032d0 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:49.36142534 +0000 UTC m=+30.673824607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/84a754a9-1c06-4302-bb01-4f53a1e032d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vv2qm" (UID: "84a754a9-1c06-4302-bb01-4f53a1e032d0") : configmap "kube-rbac-proxy" not found
Oct 31 00:18:48.861587 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.861478    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q65jt\" (UniqueName: \"kubernetes.io/projected/8072caf6-1730-4100-a87a-630ace22a370-kube-api-access-q65jt\") pod \"ingress-canary-h26l7\" (UID: \"8072caf6-1730-4100-a87a-630ace22a370\") " pod="openshift-ingress-canary/ingress-canary-h26l7"
Oct 31 00:18:48.861587 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.861504    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvblv\" (UniqueName: \"kubernetes.io/projected/f18d4e41-daf0-4be5-bcdb-c00d189419ee-kube-api-access-bvblv\") pod \"service-ca-5fffc97bdf-lpntz\" (UID: \"f18d4e41-daf0-4be5-bcdb-c00d189419ee\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpntz"
Oct 31 00:18:48.861587 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.861532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/79ab34ed-5163-4cff-8cf4-170c3445e655-metrics-tls\") pod \"dns-default-v8ft8\" (UID: \"79ab34ed-5163-4cff-8cf4-170c3445e655\") " pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:18:48.861863 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.861583    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f18d4e41-daf0-4be5-bcdb-c00d189419ee-signing-cabundle\") pod \"service-ca-5fffc97bdf-lpntz\" (UID: \"f18d4e41-daf0-4be5-bcdb-c00d189419ee\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpntz"
Oct 31 00:18:48.861863 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:48.861603    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:18:48.861863 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:48.861637    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79ab34ed-5163-4cff-8cf4-170c3445e655-metrics-tls podName:79ab34ed-5163-4cff-8cf4-170c3445e655 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:49.361625826 +0000 UTC m=+30.674025091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/79ab34ed-5163-4cff-8cf4-170c3445e655-metrics-tls") pod "dns-default-v8ft8" (UID: "79ab34ed-5163-4cff-8cf4-170c3445e655") : secret "dns-default-metrics-tls" not found
Oct 31 00:18:48.861863 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.861734    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84a754a9-1c06-4302-bb01-4f53a1e032d0-data-volume\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:18:48.861863 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.861855    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/79ab34ed-5163-4cff-8cf4-170c3445e655-config-volume\") pod \"dns-default-v8ft8\" (UID: \"79ab34ed-5163-4cff-8cf4-170c3445e655\") " pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:18:48.862013 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.861924    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79ab34ed-5163-4cff-8cf4-170c3445e655-tmp-dir\") pod \"dns-default-v8ft8\" (UID: \"79ab34ed-5163-4cff-8cf4-170c3445e655\") " pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:18:48.865048 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.865024    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8072caf6-1730-4100-a87a-630ace22a370-cert\") pod \"ingress-canary-h26l7\" (UID: \"8072caf6-1730-4100-a87a-630ace22a370\") " pod="openshift-ingress-canary/ingress-canary-h26l7"
Oct 31 00:18:48.890303 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.890274    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q65jt\" (UniqueName: \"kubernetes.io/projected/8072caf6-1730-4100-a87a-630ace22a370-kube-api-access-q65jt\") pod \"ingress-canary-h26l7\" (UID: \"8072caf6-1730-4100-a87a-630ace22a370\") " pod="openshift-ingress-canary/ingress-canary-h26l7"
Oct 31 00:18:48.890932 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.890909    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvxxw\" (UniqueName: \"kubernetes.io/projected/79ab34ed-5163-4cff-8cf4-170c3445e655-kube-api-access-jvxxw\") pod \"dns-default-v8ft8\" (UID: \"79ab34ed-5163-4cff-8cf4-170c3445e655\") " pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:18:48.892856 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.892836    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwvkk\" (UniqueName: \"kubernetes.io/projected/84a754a9-1c06-4302-bb01-4f53a1e032d0-kube-api-access-fwvkk\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:18:48.962243 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.962075    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f18d4e41-daf0-4be5-bcdb-c00d189419ee-signing-key\") pod \"service-ca-5fffc97bdf-lpntz\" (UID: \"f18d4e41-daf0-4be5-bcdb-c00d189419ee\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpntz"
Oct 31 00:18:48.962388 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.962268    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvblv\" (UniqueName: \"kubernetes.io/projected/f18d4e41-daf0-4be5-bcdb-c00d189419ee-kube-api-access-bvblv\") pod \"service-ca-5fffc97bdf-lpntz\" (UID: \"f18d4e41-daf0-4be5-bcdb-c00d189419ee\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpntz"
Oct 31 00:18:48.962388 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.962298    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f18d4e41-daf0-4be5-bcdb-c00d189419ee-signing-cabundle\") pod \"service-ca-5fffc97bdf-lpntz\" (UID: \"f18d4e41-daf0-4be5-bcdb-c00d189419ee\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpntz"
Oct 31 00:18:48.963002 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.962979    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f18d4e41-daf0-4be5-bcdb-c00d189419ee-signing-cabundle\") pod \"service-ca-5fffc97bdf-lpntz\" (UID: \"f18d4e41-daf0-4be5-bcdb-c00d189419ee\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpntz"
Oct 31 00:18:48.964770 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.964747    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f18d4e41-daf0-4be5-bcdb-c00d189419ee-signing-key\") pod \"service-ca-5fffc97bdf-lpntz\" (UID: \"f18d4e41-daf0-4be5-bcdb-c00d189419ee\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpntz"
Oct 31 00:18:48.989246 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.989216    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvblv\" (UniqueName: \"kubernetes.io/projected/f18d4e41-daf0-4be5-bcdb-c00d189419ee-kube-api-access-bvblv\") pod \"service-ca-5fffc97bdf-lpntz\" (UID: \"f18d4e41-daf0-4be5-bcdb-c00d189419ee\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpntz"
Oct 31 00:18:48.992069 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:48.992041    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h26l7"
Oct 31 00:18:49.067585 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:49.067543    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-lpntz"
Oct 31 00:18:49.366609 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:49.366579    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/79ab34ed-5163-4cff-8cf4-170c3445e655-metrics-tls\") pod \"dns-default-v8ft8\" (UID: \"79ab34ed-5163-4cff-8cf4-170c3445e655\") " pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:18:49.367141 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:49.366636    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84a754a9-1c06-4302-bb01-4f53a1e032d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:18:49.367141 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:49.366700    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84a754a9-1c06-4302-bb01-4f53a1e032d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:18:49.367141 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:49.366763    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:18:49.367141 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:49.366777    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 00:18:49.367141 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:49.366806    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:18:49.367141 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:49.366831    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79ab34ed-5163-4cff-8cf4-170c3445e655-metrics-tls podName:79ab34ed-5163-4cff-8cf4-170c3445e655 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:50.366816928 +0000 UTC m=+31.679216182 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/79ab34ed-5163-4cff-8cf4-170c3445e655-metrics-tls") pod "dns-default-v8ft8" (UID: "79ab34ed-5163-4cff-8cf4-170c3445e655") : secret "dns-default-metrics-tls" not found
Oct 31 00:18:49.367141 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:49.366846    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84a754a9-1c06-4302-bb01-4f53a1e032d0-kube-rbac-proxy-cm podName:84a754a9-1c06-4302-bb01-4f53a1e032d0 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:50.366841188 +0000 UTC m=+31.679240438 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/84a754a9-1c06-4302-bb01-4f53a1e032d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vv2qm" (UID: "84a754a9-1c06-4302-bb01-4f53a1e032d0") : configmap "kube-rbac-proxy" not found
Oct 31 00:18:49.367141 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:49.366859    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84a754a9-1c06-4302-bb01-4f53a1e032d0-insights-runtime-extractor-tls podName:84a754a9-1c06-4302-bb01-4f53a1e032d0 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:50.366852757 +0000 UTC m=+31.679252007 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84a754a9-1c06-4302-bb01-4f53a1e032d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vv2qm" (UID: "84a754a9-1c06-4302-bb01-4f53a1e032d0") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:18:49.399302 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:49.399277    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-lpntz"]
Oct 31 00:18:49.400786 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:49.400762    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf18d4e41_daf0_4be5_bcdb_c00d189419ee.slice/crio-cc3c27ce2cac021f57f143020d30cb0b82170126b74b7091267f0c5746542114 WatchSource:0}: Error finding container cc3c27ce2cac021f57f143020d30cb0b82170126b74b7091267f0c5746542114: Status 404 returned error can't find the container with id cc3c27ce2cac021f57f143020d30cb0b82170126b74b7091267f0c5746542114
Oct 31 00:18:49.402290 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:49.402268    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h26l7"]
Oct 31 00:18:49.402382 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:49.402314    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8072caf6_1730_4100_a87a_630ace22a370.slice/crio-033e943c5f2a00f0e0cd21ab0644ebef297bab6faa89eb64c4b9364c6d1900e3 WatchSource:0}: Error finding container 033e943c5f2a00f0e0cd21ab0644ebef297bab6faa89eb64c4b9364c6d1900e3: Status 404 returned error can't find the container with id 033e943c5f2a00f0e0cd21ab0644ebef297bab6faa89eb64c4b9364c6d1900e3
Oct 31 00:18:49.474305 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:49.474264    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h26l7" event={"ID":"8072caf6-1730-4100-a87a-630ace22a370","Type":"ContainerStarted","Data":"033e943c5f2a00f0e0cd21ab0644ebef297bab6faa89eb64c4b9364c6d1900e3"}
Oct 31 00:18:49.475335 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:49.475310    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-lpntz" event={"ID":"f18d4e41-daf0-4be5-bcdb-c00d189419ee","Type":"ContainerStarted","Data":"cc3c27ce2cac021f57f143020d30cb0b82170126b74b7091267f0c5746542114"}
Oct 31 00:18:49.639917 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:49.639846    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9d69r_ec6a6238-81be-4a82-b346-b4c9fe3aefa4/dns-node-resolver/0.log"
Oct 31 00:18:49.957587 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:49.956188    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:49.957587 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:49.956478    2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 00:18:49.975278 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:49.975249    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:18:50.197789 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:50.197757    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:50.197789 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:50.197779    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:50.197789 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:50.197789    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:50.216758 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:50.216731    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 00:18:50.216848 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:50.216839    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 00:18:50.217310 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:50.217296    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 00:18:50.217416 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:50.217398    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9wq7f\""
Oct 31 00:18:50.217416 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:50.217410    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-6b92m\""
Oct 31 00:18:50.217514 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:50.217457    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dsrzd\""
Oct 31 00:18:50.217968 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:50.217945    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 00:18:50.378103 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:50.378064    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84a754a9-1c06-4302-bb01-4f53a1e032d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:18:50.378775 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:50.378155    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/79ab34ed-5163-4cff-8cf4-170c3445e655-metrics-tls\") pod \"dns-default-v8ft8\" (UID: \"79ab34ed-5163-4cff-8cf4-170c3445e655\") " pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:18:50.378775 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:50.378176    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 00:18:50.378775 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:50.378221    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84a754a9-1c06-4302-bb01-4f53a1e032d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:18:50.378775 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:50.378258    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84a754a9-1c06-4302-bb01-4f53a1e032d0-kube-rbac-proxy-cm podName:84a754a9-1c06-4302-bb01-4f53a1e032d0 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:52.378236883 +0000 UTC m=+33.690636148 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/84a754a9-1c06-4302-bb01-4f53a1e032d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vv2qm" (UID: "84a754a9-1c06-4302-bb01-4f53a1e032d0") : configmap "kube-rbac-proxy" not found
Oct 31 00:18:50.378775 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:50.378323    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:18:50.378775 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:50.378324    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:18:50.378775 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:50.378377    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84a754a9-1c06-4302-bb01-4f53a1e032d0-insights-runtime-extractor-tls podName:84a754a9-1c06-4302-bb01-4f53a1e032d0 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:52.378360055 +0000 UTC m=+33.690759325 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84a754a9-1c06-4302-bb01-4f53a1e032d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vv2qm" (UID: "84a754a9-1c06-4302-bb01-4f53a1e032d0") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:18:50.378775 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:50.378395    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79ab34ed-5163-4cff-8cf4-170c3445e655-metrics-tls podName:79ab34ed-5163-4cff-8cf4-170c3445e655 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:52.37838556 +0000 UTC m=+33.690784815 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/79ab34ed-5163-4cff-8cf4-170c3445e655-metrics-tls") pod "dns-default-v8ft8" (UID: "79ab34ed-5163-4cff-8cf4-170c3445e655") : secret "dns-default-metrics-tls" not found
Oct 31 00:18:50.572894 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:50.572824    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-htbtd_95f3be30-2d65-4373-9623-155c841dca11/node-ca/0.log"
Oct 31 00:18:52.393867 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:52.393837    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84a754a9-1c06-4302-bb01-4f53a1e032d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:18:52.394365 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:52.393896    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/79ab34ed-5163-4cff-8cf4-170c3445e655-metrics-tls\") pod \"dns-default-v8ft8\" (UID: \"79ab34ed-5163-4cff-8cf4-170c3445e655\") " pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:18:52.394365 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:52.393953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84a754a9-1c06-4302-bb01-4f53a1e032d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:18:52.394365 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:52.394073    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:18:52.394365 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:52.394084    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:18:52.394365 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:52.394146    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84a754a9-1c06-4302-bb01-4f53a1e032d0-insights-runtime-extractor-tls podName:84a754a9-1c06-4302-bb01-4f53a1e032d0 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:56.394125349 +0000 UTC m=+37.706524601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84a754a9-1c06-4302-bb01-4f53a1e032d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vv2qm" (UID: "84a754a9-1c06-4302-bb01-4f53a1e032d0") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:18:52.394365 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:52.394164    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79ab34ed-5163-4cff-8cf4-170c3445e655-metrics-tls podName:79ab34ed-5163-4cff-8cf4-170c3445e655 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:56.39415543 +0000 UTC m=+37.706554692 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/79ab34ed-5163-4cff-8cf4-170c3445e655-metrics-tls") pod "dns-default-v8ft8" (UID: "79ab34ed-5163-4cff-8cf4-170c3445e655") : secret "dns-default-metrics-tls" not found
Oct 31 00:18:52.394687 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:52.394657    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84a754a9-1c06-4302-bb01-4f53a1e032d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:18:52.483723 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:52.483668    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-lpntz" event={"ID":"f18d4e41-daf0-4be5-bcdb-c00d189419ee","Type":"ContainerStarted","Data":"51962af4f85f39aa3cb5d7e8a3f52a1645eef6cd4dd04b8eee4d1dce3a064227"}
Oct 31 00:18:52.485424 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:52.485399    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h26l7" event={"ID":"8072caf6-1730-4100-a87a-630ace22a370","Type":"ContainerStarted","Data":"ff5fe9302992dcdabc75875b88a5cb1d9e9bd803da4d3469ec2662d4216821c0"}
Oct 31 00:18:52.531450 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:52.531403    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-lpntz" podStartSLOduration=1.9133909949999999 podStartE2EDuration="4.531389977s" totalImagesPullingTime="2.617998982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:52.530697191 +0000 UTC m=+33.843096458" watchObservedRunningTime="2026-10-31 00:18:52.531389977 +0000 UTC m=+33.843789248"
Oct 31 00:18:52.588136 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:52.588003    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-h26l7" podStartSLOduration=1.742359978 podStartE2EDuration="4.587988903s" totalImagesPullingTime="2.845628925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:52.587497821 +0000 UTC m=+33.899897094" watchObservedRunningTime="2026-10-31 00:18:52.587988903 +0000 UTC m=+33.900388174"
Oct 31 00:18:52.998356 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:52.998318    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-metrics-certs\") pod \"network-metrics-daemon-5tw4x\" (UID: \"3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a\") " pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:18:52.998542 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:52.998376    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/002d2da2-bb18-4667-98c8-1770b70d9985-original-pull-secret\") pod \"global-pull-secret-syncer-pncsr\" (UID: \"002d2da2-bb18-4667-98c8-1770b70d9985\") " pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:52.998542 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:52.998509    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 00:18:52.998664 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:52.998589    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-metrics-certs podName:3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a nodeName:}" failed. No retries permitted until 2026-10-31 00:19:24.998569287 +0000 UTC m=+66.310968552 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-metrics-certs") pod "network-metrics-daemon-5tw4x" (UID: "3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a") : secret "metrics-daemon-secret" not found
Oct 31 00:18:53.001807 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:53.001784    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/002d2da2-bb18-4667-98c8-1770b70d9985-original-pull-secret\") pod \"global-pull-secret-syncer-pncsr\" (UID: \"002d2da2-bb18-4667-98c8-1770b70d9985\") " pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:53.099272 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:53.099228    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngkbc\" (UniqueName: \"kubernetes.io/projected/5b9f4a03-68eb-418e-9447-ad62473a1e9a-kube-api-access-ngkbc\") pod \"network-check-target-vdxch\" (UID: \"5b9f4a03-68eb-418e-9447-ad62473a1e9a\") " pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:53.105322 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:53.105289    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngkbc\" (UniqueName: \"kubernetes.io/projected/5b9f4a03-68eb-418e-9447-ad62473a1e9a-kube-api-access-ngkbc\") pod \"network-check-target-vdxch\" (UID: \"5b9f4a03-68eb-418e-9447-ad62473a1e9a\") " pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:53.234751 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:53.234713    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:53.252579 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:53.252505    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:18:53.581957 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:53.581934    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vdxch"]
Oct 31 00:18:53.583989 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:53.583963    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b9f4a03_68eb_418e_9447_ad62473a1e9a.slice/crio-413933e427d61a4d7788dd590ddd013dd2af5d6a1d89cad636cca4365cc875b8 WatchSource:0}: Error finding container 413933e427d61a4d7788dd590ddd013dd2af5d6a1d89cad636cca4365cc875b8: Status 404 returned error can't find the container with id 413933e427d61a4d7788dd590ddd013dd2af5d6a1d89cad636cca4365cc875b8
Oct 31 00:18:53.615309 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:53.615275    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pncsr"]
Oct 31 00:18:53.616235 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:53.616215    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod002d2da2_bb18_4667_98c8_1770b70d9985.slice/crio-93de93df25f639d3b26ba4c93369d11af58424ab46e708468f2273c84b6b7cb8 WatchSource:0}: Error finding container 93de93df25f639d3b26ba4c93369d11af58424ab46e708468f2273c84b6b7cb8: Status 404 returned error can't find the container with id 93de93df25f639d3b26ba4c93369d11af58424ab46e708468f2273c84b6b7cb8
Oct 31 00:18:54.492998 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:54.492919    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vdxch" event={"ID":"5b9f4a03-68eb-418e-9447-ad62473a1e9a","Type":"ContainerStarted","Data":"413933e427d61a4d7788dd590ddd013dd2af5d6a1d89cad636cca4365cc875b8"}
Oct 31 00:18:54.494139 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:54.494111    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pncsr" event={"ID":"002d2da2-bb18-4667-98c8-1770b70d9985","Type":"ContainerStarted","Data":"93de93df25f639d3b26ba4c93369d11af58424ab46e708468f2273c84b6b7cb8"}
Oct 31 00:18:56.427457 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:56.427410    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84a754a9-1c06-4302-bb01-4f53a1e032d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:18:56.427988 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:56.427486    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/79ab34ed-5163-4cff-8cf4-170c3445e655-metrics-tls\") pod \"dns-default-v8ft8\" (UID: \"79ab34ed-5163-4cff-8cf4-170c3445e655\") " pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:18:56.427988 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:56.427585    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:18:56.427988 ip-10-0-128-253 kubenswrapper[2580]: E1031 00:18:56.427666    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84a754a9-1c06-4302-bb01-4f53a1e032d0-insights-runtime-extractor-tls podName:84a754a9-1c06-4302-bb01-4f53a1e032d0 nodeName:}" failed. No retries permitted until 2026-10-31 00:19:04.427644793 +0000 UTC m=+45.740044059 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84a754a9-1c06-4302-bb01-4f53a1e032d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vv2qm" (UID: "84a754a9-1c06-4302-bb01-4f53a1e032d0") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:18:56.431808 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:56.431775    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/79ab34ed-5163-4cff-8cf4-170c3445e655-metrics-tls\") pod \"dns-default-v8ft8\" (UID: \"79ab34ed-5163-4cff-8cf4-170c3445e655\") " pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:18:56.501850 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:56.501816    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:18:56.843935 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:56.843709    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v8ft8"]
Oct 31 00:18:56.846415 ip-10-0-128-253 kubenswrapper[2580]: W1031 00:18:56.846382    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ab34ed_5163_4cff_8cf4_170c3445e655.slice/crio-a5a25be85fe813b43a6e7973d872017c0d5b8552f078b3a1c0eac4df30054064 WatchSource:0}: Error finding container a5a25be85fe813b43a6e7973d872017c0d5b8552f078b3a1c0eac4df30054064: Status 404 returned error can't find the container with id a5a25be85fe813b43a6e7973d872017c0d5b8552f078b3a1c0eac4df30054064
Oct 31 00:18:57.505182 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:57.505144    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vdxch" event={"ID":"5b9f4a03-68eb-418e-9447-ad62473a1e9a","Type":"ContainerStarted","Data":"3c83805f74091faa33d83a04dcbd60d2e834fc488944f0b285cd140defd5c984"}
Oct 31 00:18:57.505883 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:57.505267    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:18:57.507011 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:57.506984    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v8ft8" event={"ID":"79ab34ed-5163-4cff-8cf4-170c3445e655","Type":"ContainerStarted","Data":"a5a25be85fe813b43a6e7973d872017c0d5b8552f078b3a1c0eac4df30054064"}
Oct 31 00:18:59.517353 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:59.517165    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pncsr" event={"ID":"002d2da2-bb18-4667-98c8-1770b70d9985","Type":"ContainerStarted","Data":"766ce34fc6903867cf72e2b21d3f6981d372568d937f7e61a1d7028eb68c2b91"}
Oct 31 00:18:59.521154 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:59.521120    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v8ft8" event={"ID":"79ab34ed-5163-4cff-8cf4-170c3445e655","Type":"ContainerStarted","Data":"731943d06b934422905abf662bfd614fa01e7a76a3477511ef8cdfe15eda67bb"}
Oct 31 00:18:59.563417 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:59.563261    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pncsr" podStartSLOduration=34.822867485 podStartE2EDuration="40.563243435s" totalImagesPullingTime="5.74037595s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:59.562353438 +0000 UTC m=+40.874752709" watchObservedRunningTime="2026-10-31 00:18:59.563243435 +0000 UTC m=+40.875642708"
Oct 31 00:18:59.563577 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:59.563423    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vdxch" podStartSLOduration=36.83763737 podStartE2EDuration="40.563416644s" totalImagesPullingTime="3.725779274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:57.552942766 +0000 UTC m=+38.865342037" watchObservedRunningTime="2026-10-31 00:18:59.563416644 +0000 UTC m=+40.875815916"
Oct 31 00:18:59.694077 ip-10-0-128-253 kubenswrapper[2580]: I1031 00:18:59.693962    2580 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 00:18:59.694129 ip-10-0-128-253 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 00:18:59.708401 ip-10-0-128-253 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 00:18:59.708580 ip-10-0-128-253 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 00:18:59.708774 ip-10-0-128-253 systemd[1]: kubelet.service: Consumed 2.487s CPU time.
Oct 31 00:18:59.720331 ip-10-0-128-253 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 00:18:59.836069 ip-10-0-128-253 kubenswrapper[4327]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:18:59.836069 ip-10-0-128-253 kubenswrapper[4327]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 00:18:59.836069 ip-10-0-128-253 kubenswrapper[4327]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:18:59.836069 ip-10-0-128-253 kubenswrapper[4327]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:18:59.839149 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839066    4327 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:18:59.839149 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839145    4327 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:18:59.839149 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839150    4327 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:18:59.839149 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839153    4327 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:18:59.839149 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839157    4327 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:18:59.839338 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839160    4327 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:18:59.839338 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839163    4327 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:18:59.839338 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839166    4327 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:18:59.839338 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839169    4327 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:18:59.839338 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839171    4327 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:18:59.839338 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839174    4327 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:18:59.839338 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839177    4327 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:18:59.839338 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839180    4327 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:18:59.839338 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839182    4327 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:18:59.839338 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839185    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:18:59.839338 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839188    4327 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:18:59.839338 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839190    4327 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:18:59.839338 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839193    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:18:59.839338 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839196    4327 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:18:59.839338 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839199    4327 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:18:59.839338 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839202    4327 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:18:59.839338 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839204    4327 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:18:59.839338 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839207    4327 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:18:59.839338 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839209    4327 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:18:59.839834 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839212    4327 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:18:59.839834 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839216    4327 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:18:59.839834 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839218    4327 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:18:59.839834 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839222    4327 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:18:59.839834 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839225    4327 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:18:59.839834 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839227    4327 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:18:59.839834 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839237    4327 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:18:59.839834 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839240    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:18:59.839834 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839243    4327 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:18:59.839834 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839246    4327 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:18:59.839834 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839249    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:18:59.839834 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839252    4327 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:18:59.839834 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839255    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:18:59.839834 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839258    4327 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:18:59.839834 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839261    4327 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:18:59.839834 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839264    4327 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:18:59.839834 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839266    4327 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:18:59.839834 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839269    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:18:59.839834 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839272    4327 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:18:59.840295 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839275    4327 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:18:59.840295 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839278    4327 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:18:59.840295 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839281    4327 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:18:59.840295 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839283    4327 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:18:59.840295 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839286    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:18:59.840295 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839288    4327 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:18:59.840295 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839291    4327 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:18:59.840295 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839295    4327 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:18:59.840295 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839297    4327 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:18:59.840295 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839300    4327 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:18:59.840295 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839303    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:18:59.840295 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839306    4327 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:18:59.840295 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839309    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:18:59.840295 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839311    4327 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:18:59.840295 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839314    4327 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:18:59.840295 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839317    4327 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:18:59.840295 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839319    4327 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:18:59.840295 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839322    4327 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:18:59.840295 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839325    4327 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:18:59.840778 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839327    4327 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:18:59.840778 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839330    4327 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:18:59.840778 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839332    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:18:59.840778 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839336    4327 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:18:59.840778 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839339    4327 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:18:59.840778 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839341    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:18:59.840778 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839344    4327 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:18:59.840778 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839347    4327 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:18:59.840778 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839350    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:18:59.840778 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839353    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:18:59.840778 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839355    4327 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:18:59.840778 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839359    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:18:59.840778 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839361    4327 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:18:59.840778 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839364    4327 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:18:59.840778 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839367    4327 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:18:59.840778 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839370    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:18:59.840778 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839373    4327 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:18:59.840778 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839376    4327 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:18:59.840778 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839379    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:18:59.840778 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839381    4327 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:18:59.841259 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839384    4327 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:18:59.841259 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839387    4327 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:18:59.841259 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839390    4327 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:18:59.841259 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839393    4327 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:18:59.841259 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839395    4327 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:18:59.841259 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839398    4327 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:18:59.841259 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839401    4327 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:18:59.841259 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839403    4327 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:18:59.841259 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839407    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:18:59.841259 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839410    4327 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:18:59.841259 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839412    4327 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:18:59.841259 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839417    4327 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:18:59.841259 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839421    4327 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:18:59.841259 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839424    4327 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:18:59.841259 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839427    4327 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:18:59.841259 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839431    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:18:59.841259 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839434    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:18:59.841259 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839437    4327 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:18:59.841259 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839440    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:18:59.841259 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839443    4327 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:18:59.841756 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839445    4327 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:18:59.841756 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839450    4327 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:18:59.841756 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839453    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:18:59.841756 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.839456    4327 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:18:59.841756 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840287    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:18:59.841756 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840295    4327 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:18:59.841756 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840299    4327 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:18:59.841756 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840302    4327 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:18:59.841756 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840305    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:18:59.841756 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840308    4327 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:18:59.841756 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840312    4327 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:18:59.841756 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840317    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:18:59.841756 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840320    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:18:59.841756 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840323    4327 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:18:59.841756 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840326    4327 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:18:59.841756 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840330    4327 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:18:59.841756 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840333    4327 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:18:59.841756 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840336    4327 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:18:59.841756 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840338    4327 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:18:59.842217 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840341    4327 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:18:59.842217 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840344    4327 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:18:59.842217 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840346    4327 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:18:59.842217 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840349    4327 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:18:59.842217 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840352    4327 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:18:59.842217 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840354    4327 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:18:59.842217 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840357    4327 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:18:59.842217 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840360    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:18:59.842217 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840363    4327 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:18:59.842217 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840366    4327 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:18:59.842217 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840369    4327 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:18:59.842217 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840372    4327 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:18:59.842217 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840374    4327 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:18:59.842217 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840377    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:18:59.842217 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840380    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:18:59.842217 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840383    4327 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:18:59.842217 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840385    4327 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:18:59.842217 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840388    4327 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:18:59.842217 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840391    4327 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:18:59.842217 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840393    4327 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:18:59.842715 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840396    4327 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:18:59.842715 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840399    4327 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:18:59.842715 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840402    4327 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:18:59.842715 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840404    4327 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:18:59.842715 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840407    4327 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:18:59.842715 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840410    4327 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:18:59.842715 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840412    4327 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:18:59.842715 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840415    4327 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:18:59.842715 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840418    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:18:59.842715 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840420    4327 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:18:59.842715 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840423    4327 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:18:59.842715 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840426    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:18:59.842715 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840428    4327 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:18:59.842715 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840431    4327 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:18:59.842715 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840434    4327 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:18:59.842715 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840436    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:18:59.842715 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840439    4327 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:18:59.842715 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840442    4327 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:18:59.842715 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840444    4327 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:18:59.842715 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840447    4327 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:18:59.843196 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840450    4327 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:18:59.843196 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840453    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:18:59.843196 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840456    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:18:59.843196 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840460    4327 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:18:59.843196 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840464    4327 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:18:59.843196 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840467    4327 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:18:59.843196 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840470    4327 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:18:59.843196 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840473    4327 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:18:59.843196 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840475    4327 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:18:59.843196 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840478    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:18:59.843196 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840481    4327 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:18:59.843196 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840483    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:18:59.843196 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840486    4327 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:18:59.843196 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840489    4327 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:18:59.843196 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840492    4327 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:18:59.843196 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840494    4327 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:18:59.843196 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840497    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:18:59.843196 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840499    4327 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:18:59.843196 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840502    4327 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:18:59.843699 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840505    4327 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:18:59.843699 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840507    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:18:59.843699 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840511    4327 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:18:59.843699 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840514    4327 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:18:59.843699 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840516    4327 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:18:59.843699 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840519    4327 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:18:59.843699 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840521    4327 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:18:59.843699 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840524    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:18:59.843699 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840527    4327 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:18:59.843699 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840529    4327 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:18:59.843699 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840533    4327 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:18:59.843699 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840535    4327 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:18:59.843699 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840538    4327 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:18:59.843699 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840545    4327 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:18:59.843699 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840548    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:18:59.843699 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840550    4327 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:18:59.843699 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840553    4327 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:18:59.843699 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840556    4327 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:18:59.843699 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840558    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:18:59.843699 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840561    4327 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:18:59.844192 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840564    4327 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:18:59.844192 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840566    4327 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:18:59.844192 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840569    4327 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:18:59.844192 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840571    4327 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:18:59.844192 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840574    4327 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:18:59.844192 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840577    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:18:59.844192 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840579    4327 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:18:59.844192 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840582    4327 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:18:59.844192 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840584    4327 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:18:59.844192 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840587    4327 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:18:59.844192 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840591    4327 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:18:59.844192 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.840593    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:18:59.844192 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841088    4327 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:18:59.844192 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841094    4327 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:18:59.844192 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841098    4327 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:18:59.844192 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841101    4327 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:18:59.844192 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841104    4327 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:18:59.844192 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841106    4327 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:18:59.844192 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841112    4327 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:18:59.844651 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841114    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:18:59.844651 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841117    4327 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:18:59.844651 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841120    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:18:59.844651 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841122    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:18:59.844651 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841126    4327 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:18:59.844651 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841128    4327 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:18:59.844651 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841132    4327 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:18:59.844651 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841135    4327 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:18:59.844651 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841137    4327 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:18:59.844651 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841140    4327 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:18:59.844651 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841142    4327 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:18:59.844651 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841145    4327 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:18:59.844651 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841148    4327 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:18:59.844651 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841151    4327 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:18:59.844651 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841153    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:18:59.844651 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841157    4327 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:18:59.844651 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841164    4327 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:18:59.844651 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841167    4327 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:18:59.844651 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841170    4327 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:18:59.845119 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841173    4327 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:18:59.845119 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841176    4327 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:18:59.845119 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841179    4327 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:18:59.845119 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841181    4327 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:18:59.845119 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841184    4327 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:18:59.845119 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841187    4327 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:18:59.845119 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841189    4327 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:18:59.845119 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841192    4327 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:18:59.845119 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841195    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:18:59.845119 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841198    4327 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:18:59.845119 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841201    4327 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:18:59.845119 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841204    4327 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:18:59.845119 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841208    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:18:59.845119 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841211    4327 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:18:59.845119 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841214    4327 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:18:59.845119 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841217    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:18:59.845119 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841219    4327 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:18:59.845119 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841222    4327 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:18:59.845119 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841225    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:18:59.845119 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841229    4327 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:18:59.845594 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841232    4327 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:18:59.845594 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841235    4327 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:18:59.845594 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841238    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:18:59.845594 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841240    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:18:59.845594 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841243    4327 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:18:59.845594 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841246    4327 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:18:59.845594 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841249    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:18:59.845594 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841252    4327 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:18:59.845594 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841255    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:18:59.845594 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841257    4327 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:18:59.845594 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841260    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:18:59.845594 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841263    4327 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:18:59.845594 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841266    4327 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:18:59.845594 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841269    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:18:59.845594 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841272    4327 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:18:59.845594 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841275    4327 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:18:59.845594 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841278    4327 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:18:59.845594 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841280    4327 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:18:59.845594 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841283    4327 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:18:59.845594 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841286    4327 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:18:59.846100 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841288    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:18:59.846100 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841291    4327 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:18:59.846100 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841295    4327 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:18:59.846100 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841298    4327 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:18:59.846100 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841302    4327 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:18:59.846100 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841304    4327 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:18:59.846100 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841308    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:18:59.846100 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841311    4327 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:18:59.846100 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841314    4327 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:18:59.846100 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841317    4327 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:18:59.846100 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841319    4327 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:18:59.846100 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841324    4327 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:18:59.846100 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841327    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:18:59.846100 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841330    4327 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:18:59.846100 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841332    4327 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:18:59.846100 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841335    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:18:59.846100 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841337    4327 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:18:59.846100 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841340    4327 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:18:59.846100 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841344    4327 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:18:59.846818 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841349    4327 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:18:59.846818 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841352    4327 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:18:59.846818 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841355    4327 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:18:59.846818 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841358    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:18:59.846818 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841360    4327 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:18:59.846818 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841363    4327 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:18:59.846818 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841366    4327 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:18:59.846818 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841369    4327 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:18:59.846818 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841371    4327 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:18:59.846818 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841374    4327 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:18:59.846818 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841377    4327 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:18:59.846818 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841379    4327 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:18:59.846818 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841382    4327 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:18:59.846818 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841384    4327 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:18:59.846818 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841387    4327 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:18:59.846818 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841390    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:18:59.846818 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841393    4327 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:18:59.846818 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841397    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:18:59.846818 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841400    4327 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:18:59.846818 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841404    4327 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:18:59.847318 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.841407    4327 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:18:59.847318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841500    4327 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 00:18:59.847318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841508    4327 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 00:18:59.847318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841513    4327 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 00:18:59.847318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841520    4327 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 00:18:59.847318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841524    4327 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 00:18:59.847318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841528    4327 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 00:18:59.847318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841532    4327 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 00:18:59.847318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841536    4327 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 00:18:59.847318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841540    4327 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 00:18:59.847318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841543    4327 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 00:18:59.847318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841547    4327 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 00:18:59.847318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841550    4327 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 00:18:59.847318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841553    4327 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 00:18:59.847318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841556    4327 flags.go:64] FLAG: --cgroup-root=""
Oct 31 00:18:59.847318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841559    4327 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 00:18:59.847318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841562    4327 flags.go:64] FLAG: --client-ca-file=""
Oct 31 00:18:59.847318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841565    4327 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 00:18:59.847318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841569    4327 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 00:18:59.847318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841574    4327 flags.go:64] FLAG: --cluster-domain=""
Oct 31 00:18:59.847318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841577    4327 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 00:18:59.847318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841580    4327 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 00:18:59.847318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841583    4327 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 00:18:59.847318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841586    4327 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 00:18:59.847911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841595    4327 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 00:18:59.847911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841598    4327 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 00:18:59.847911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841601    4327 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 00:18:59.847911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841604    4327 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 00:18:59.847911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841607    4327 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 00:18:59.847911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841617    4327 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 00:18:59.847911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841620    4327 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 00:18:59.847911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841623    4327 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 00:18:59.847911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841626    4327 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 00:18:59.847911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841631    4327 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 00:18:59.847911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841634    4327 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 00:18:59.847911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841637    4327 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 00:18:59.847911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841641    4327 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 00:18:59.847911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841645    4327 flags.go:64] FLAG: --enable-server="true"
Oct 31 00:18:59.847911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841648    4327 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 00:18:59.847911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841652    4327 flags.go:64] FLAG: --event-burst="100"
Oct 31 00:18:59.847911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841655    4327 flags.go:64] FLAG: --event-qps="50"
Oct 31 00:18:59.847911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841658    4327 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 00:18:59.847911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841662    4327 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 00:18:59.847911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841665    4327 flags.go:64] FLAG: --eviction-hard=""
Oct 31 00:18:59.847911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841669    4327 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 00:18:59.847911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841672    4327 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 00:18:59.847911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841695    4327 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 00:18:59.847911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841701    4327 flags.go:64] FLAG: --eviction-soft=""
Oct 31 00:18:59.847911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841704    4327 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 00:18:59.848570 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841707    4327 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 00:18:59.848570 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841710    4327 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 00:18:59.848570 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841713    4327 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 00:18:59.848570 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841716    4327 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 00:18:59.848570 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841719    4327 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 00:18:59.848570 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841722    4327 flags.go:64] FLAG: --feature-gates=""
Oct 31 00:18:59.848570 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841727    4327 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 00:18:59.848570 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841730    4327 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 00:18:59.848570 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841733    4327 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 00:18:59.848570 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841736    4327 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 00:18:59.848570 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841740    4327 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 00:18:59.848570 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841743    4327 flags.go:64] FLAG: --help="false"
Oct 31 00:18:59.848570 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841747    4327 flags.go:64] FLAG: --hostname-override="ip-10-0-128-253.ec2.internal"
Oct 31 00:18:59.848570 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841751    4327 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 00:18:59.848570 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841755    4327 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 00:18:59.848570 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841758    4327 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 00:18:59.848570 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841762    4327 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 00:18:59.848570 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841765    4327 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 00:18:59.848570 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841768    4327 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 00:18:59.848570 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841772    4327 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 00:18:59.848570 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841775    4327 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 00:18:59.848570 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841778    4327 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 00:18:59.848570 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841781    4327 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 00:18:59.849141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841785    4327 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 00:18:59.849141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841788    4327 flags.go:64] FLAG: --kube-reserved=""
Oct 31 00:18:59.849141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841791    4327 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 00:18:59.849141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841794    4327 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 00:18:59.849141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841797    4327 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 00:18:59.849141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841799    4327 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 00:18:59.849141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841802    4327 flags.go:64] FLAG: --lock-file=""
Oct 31 00:18:59.849141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841805    4327 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 00:18:59.849141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841808    4327 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 00:18:59.849141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841811    4327 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 00:18:59.849141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841817    4327 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 00:18:59.849141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841820    4327 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 00:18:59.849141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841824    4327 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 00:18:59.849141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841827    4327 flags.go:64] FLAG: --logging-format="text"
Oct 31 00:18:59.849141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841830    4327 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 00:18:59.849141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841834    4327 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 00:18:59.849141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841836    4327 flags.go:64] FLAG: --manifest-url=""
Oct 31 00:18:59.849141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841840    4327 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 00:18:59.849141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841844    4327 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 00:18:59.849141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841847    4327 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 00:18:59.849141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841851    4327 flags.go:64] FLAG: --max-pods="110"
Oct 31 00:18:59.849141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841855    4327 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 00:18:59.849141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841858    4327 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 00:18:59.849141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841861    4327 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 00:18:59.849141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841864    4327 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 00:18:59.849845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841867    4327 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 00:18:59.849845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841870    4327 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 00:18:59.849845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841873    4327 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 00:18:59.849845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841880    4327 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 00:18:59.849845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841883    4327 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 00:18:59.849845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841886    4327 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 00:18:59.849845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841889    4327 flags.go:64] FLAG: --pod-cidr=""
Oct 31 00:18:59.849845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841892    4327 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 00:18:59.849845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841895    4327 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 00:18:59.849845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841898    4327 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 00:18:59.849845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841901    4327 flags.go:64] FLAG: --port="10250"
Oct 31 00:18:59.849845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841904    4327 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 00:18:59.849845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841907    4327 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05ef2293b9399d368"
Oct 31 00:18:59.849845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841910    4327 flags.go:64] FLAG: --qos-reserved=""
Oct 31 00:18:59.849845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841913    4327 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 00:18:59.849845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841916    4327 flags.go:64] FLAG: --register-node="true"
Oct 31 00:18:59.849845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841919    4327 flags.go:64] FLAG: --register-with-taints=""
Oct 31 00:18:59.849845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841923    4327 flags.go:64] FLAG: --registry-burst="10"
Oct 31 00:18:59.849845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841926    4327 flags.go:64] FLAG: --registry-qps="5"
Oct 31 00:18:59.849845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841929    4327 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 00:18:59.849845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841931    4327 flags.go:64] FLAG: --reserved-memory=""
Oct 31 00:18:59.849845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841935    4327 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 00:18:59.849845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841938    4327 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 00:18:59.849845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841941    4327 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 00:18:59.849845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841944    4327 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 00:18:59.849845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841947    4327 flags.go:64] FLAG: --runonce="false"
Oct 31 00:18:59.850483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841950    4327 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 00:18:59.850483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841953    4327 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 00:18:59.850483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841957    4327 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 00:18:59.850483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841960    4327 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 00:18:59.850483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841963    4327 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 00:18:59.850483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841967    4327 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 00:18:59.850483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841970    4327 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 00:18:59.850483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841974    4327 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 00:18:59.850483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841977    4327 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 00:18:59.850483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841980    4327 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 00:18:59.850483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841983    4327 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 00:18:59.850483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841986    4327 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 00:18:59.850483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841989    4327 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 00:18:59.850483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841992    4327 flags.go:64] FLAG: --system-cgroups=""
Oct 31 00:18:59.850483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.841995    4327 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 00:18:59.850483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.842001    4327 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 00:18:59.850483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.842004    4327 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 00:18:59.850483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.842007    4327 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 00:18:59.850483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.842011    4327 flags.go:64] FLAG: --tls-min-version=""
Oct 31 00:18:59.850483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.842014    4327 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 00:18:59.850483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.842017    4327 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 00:18:59.850483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.842020    4327 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 00:18:59.850483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.842023    4327 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 00:18:59.850483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.842027    4327 flags.go:64] FLAG: --v="2"
Oct 31 00:18:59.850483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.842036    4327 flags.go:64] FLAG: --version="false"
Oct 31 00:18:59.851233 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.842040    4327 flags.go:64] FLAG: --vmodule=""
Oct 31 00:18:59.851233 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.842044    4327 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 00:18:59.851233 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.842048    4327 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 00:18:59.851233 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842191    4327 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:18:59.851233 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842195    4327 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:18:59.851233 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842198    4327 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:18:59.851233 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842201    4327 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:18:59.851233 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842204    4327 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:18:59.851233 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842207    4327 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:18:59.851233 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842209    4327 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:18:59.851233 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842212    4327 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:18:59.851233 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842215    4327 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:18:59.851233 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842217    4327 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:18:59.851233 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842221    4327 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:18:59.851233 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842224    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:18:59.851233 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842227    4327 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:18:59.851233 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842229    4327 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:18:59.851233 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842232    4327 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:18:59.851233 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842235    4327 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:18:59.851233 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842242    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:18:59.851735 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842245    4327 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:18:59.851735 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842247    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:18:59.851735 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842250    4327 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:18:59.851735 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842253    4327 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:18:59.851735 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842256    4327 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:18:59.851735 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842259    4327 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:18:59.851735 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842261    4327 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:18:59.851735 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842264    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:18:59.851735 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842266    4327 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:18:59.851735 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842269    4327 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:18:59.851735 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842272    4327 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:18:59.851735 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842275    4327 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:18:59.851735 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842278    4327 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:18:59.851735 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842280    4327 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:18:59.851735 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842283    4327 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:18:59.851735 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842286    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:18:59.851735 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842288    4327 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:18:59.851735 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842291    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:18:59.851735 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842293    4327 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:18:59.851735 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842296    4327 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:18:59.852397 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842299    4327 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:18:59.852397 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842302    4327 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:18:59.852397 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842304    4327 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:18:59.852397 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842307    4327 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:18:59.852397 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842310    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:18:59.852397 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842312    4327 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:18:59.852397 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842316    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:18:59.852397 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842319    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:18:59.852397 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842323    4327 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:18:59.852397 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842326    4327 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:18:59.852397 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842330    4327 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:18:59.852397 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842334    4327 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:18:59.852397 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842337    4327 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:18:59.852397 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842340    4327 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:18:59.852397 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842343    4327 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:18:59.852397 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842345    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:18:59.852397 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842348    4327 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:18:59.852397 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842350    4327 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:18:59.852397 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842353    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:18:59.853193 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842356    4327 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:18:59.853193 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842358    4327 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:18:59.853193 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842361    4327 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:18:59.853193 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842363    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:18:59.853193 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842366    4327 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:18:59.853193 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842373    4327 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:18:59.853193 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842375    4327 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:18:59.853193 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842379    4327 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:18:59.853193 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842383    4327 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:18:59.853193 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842387    4327 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:18:59.853193 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842390    4327 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:18:59.853193 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842393    4327 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:18:59.853193 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842396    4327 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:18:59.853193 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842399    4327 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:18:59.853193 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842402    4327 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:18:59.853193 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842405    4327 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:18:59.853193 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842409    4327 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:18:59.853193 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842411    4327 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:18:59.853193 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842414    4327 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:18:59.853692 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842417    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:18:59.853692 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842420    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:18:59.853692 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842423    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:18:59.853692 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842425    4327 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:18:59.853692 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842428    4327 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:18:59.853692 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842432    4327 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:18:59.853692 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842435    4327 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:18:59.853692 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842438    4327 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:18:59.853692 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842440    4327 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:18:59.853692 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842443    4327 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:18:59.853692 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842446    4327 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:18:59.853692 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842449    4327 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:18:59.853692 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842451    4327 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:18:59.853692 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842454    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:18:59.853692 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842457    4327 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:18:59.853692 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842459    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:18:59.853692 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842462    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:18:59.853692 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842465    4327 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:18:59.853692 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842469    4327 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:18:59.853692 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842471    4327 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:18:59.854254 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842474    4327 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:18:59.854254 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842477    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:18:59.854254 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842479    4327 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:18:59.854254 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842482    4327 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:18:59.854254 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842484    4327 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:18:59.854254 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842487    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:18:59.854254 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842490    4327 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:18:59.854254 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842493    4327 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:18:59.854254 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842495    4327 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:18:59.854254 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842498    4327 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:18:59.854254 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842501    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:18:59.854254 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.842527    4327 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:18:59.854254 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842574    4327 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:18:59.854254 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842578    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:18:59.854254 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842582    4327 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:18:59.854254 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842585    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:18:59.854645 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842588    4327 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:18:59.854645 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842592    4327 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:18:59.854645 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842594    4327 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:18:59.854645 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842597    4327 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:18:59.854645 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842600    4327 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:18:59.854645 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842602    4327 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:18:59.854645 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842605    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:18:59.854645 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842608    4327 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:18:59.854645 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842611    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:18:59.854645 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842614    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:18:59.854645 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842617    4327 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:18:59.854645 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842621    4327 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:18:59.854645 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842624    4327 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:18:59.854645 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842627    4327 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:18:59.854645 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842630    4327 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:18:59.854645 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842635    4327 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:18:59.854645 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842638    4327 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:18:59.854645 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842642    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:18:59.854645 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842645    4327 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:18:59.855215 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842647    4327 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:18:59.855215 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842650    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:18:59.855215 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842653    4327 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:18:59.855215 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842655    4327 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:18:59.855215 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842658    4327 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:18:59.855215 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842661    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:18:59.855215 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842663    4327 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:18:59.855215 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842666    4327 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:18:59.855215 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842668    4327 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:18:59.855215 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842672    4327 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:18:59.855215 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842691    4327 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:18:59.855215 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842695    4327 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:18:59.855215 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842705    4327 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:18:59.855215 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842708    4327 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:18:59.855215 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842713    4327 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:18:59.855215 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842716    4327 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:18:59.855215 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842720    4327 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:18:59.855215 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842723    4327 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:18:59.855215 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842725    4327 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:18:59.856012 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842728    4327 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:18:59.856012 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842731    4327 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:18:59.856012 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842733    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:18:59.856012 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842736    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:18:59.856012 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842739    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:18:59.856012 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842742    4327 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:18:59.856012 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842744    4327 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:18:59.856012 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842747    4327 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:18:59.856012 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842750    4327 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:18:59.856012 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842753    4327 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:18:59.856012 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842757    4327 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:18:59.856012 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842759    4327 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:18:59.856012 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842762    4327 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:18:59.856012 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842765    4327 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:18:59.856012 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842768    4327 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:18:59.856012 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842771    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:18:59.856012 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842773    4327 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:18:59.856012 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842776    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:18:59.856012 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842779    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:18:59.856012 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842781    4327 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:18:59.856796 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842784    4327 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:18:59.856796 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842787    4327 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:18:59.856796 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842789    4327 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:18:59.856796 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842792    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:18:59.856796 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842795    4327 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:18:59.856796 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842797    4327 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:18:59.856796 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842805    4327 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:18:59.856796 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842810    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:18:59.856796 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842813    4327 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:18:59.856796 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842816    4327 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:18:59.856796 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842818    4327 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:18:59.856796 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842821    4327 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:18:59.856796 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842823    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:18:59.856796 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842826    4327 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:18:59.856796 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842829    4327 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:18:59.856796 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842831    4327 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:18:59.856796 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842834    4327 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:18:59.856796 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842836    4327 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:18:59.856796 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842839    4327 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:18:59.857269 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842842    4327 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:18:59.857269 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842844    4327 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:18:59.857269 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842848    4327 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:18:59.857269 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842851    4327 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:18:59.857269 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842854    4327 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:18:59.857269 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842857    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:18:59.857269 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842859    4327 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:18:59.857269 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842862    4327 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:18:59.857269 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842864    4327 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:18:59.857269 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842867    4327 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:18:59.857269 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842870    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:18:59.857269 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842872    4327 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:18:59.857269 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842875    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:18:59.857269 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842878    4327 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:18:59.857269 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842880    4327 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:18:59.857269 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842884    4327 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:18:59.857269 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842886    4327 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:18:59.857269 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842889    4327 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:18:59.857269 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842892    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:18:59.857851 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842894    4327 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:18:59.857851 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842905    4327 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:18:59.857851 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842908    4327 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:18:59.857851 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842910    4327 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:18:59.857851 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842913    4327 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:18:59.857851 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.842916    4327 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:18:59.857851 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.848008    4327 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 00:18:59.857851 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.848022    4327 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 00:18:59.857851 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.848039    4327 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 00:18:59.857851 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.848044    4327 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 00:18:59.857851 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848100    4327 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:18:59.857851 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848105    4327 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:18:59.857851 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848109    4327 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:18:59.857851 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848112    4327 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:18:59.857851 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848115    4327 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:18:59.857851 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848117    4327 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:18:59.857851 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848120    4327 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:18:59.857851 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848123    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:18:59.857851 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848126    4327 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:18:59.857851 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848128    4327 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:18:59.858332 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848131    4327 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:18:59.858332 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848134    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:18:59.858332 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848136    4327 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:18:59.858332 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848139    4327 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:18:59.858332 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848141    4327 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:18:59.858332 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848144    4327 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:18:59.858332 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848146    4327 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:18:59.858332 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848149    4327 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:18:59.858332 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848152    4327 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:18:59.858332 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848154    4327 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:18:59.858332 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848157    4327 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:18:59.858332 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848160    4327 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:18:59.858332 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848162    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:18:59.858332 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848166    4327 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:18:59.858332 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848169    4327 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:18:59.858332 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848172    4327 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:18:59.858332 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848174    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:18:59.858332 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848177    4327 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:18:59.858332 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848180    4327 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:18:59.858332 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848183    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:18:59.858821 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848186    4327 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:18:59.858821 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848189    4327 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:18:59.858821 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848192    4327 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:18:59.858821 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848194    4327 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:18:59.858821 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848197    4327 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:18:59.858821 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848200    4327 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:18:59.858821 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848202    4327 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:18:59.858821 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848205    4327 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:18:59.858821 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848208    4327 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:18:59.858821 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848211    4327 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:18:59.858821 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848214    4327 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:18:59.858821 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848216    4327 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:18:59.858821 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848219    4327 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:18:59.858821 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848222    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:18:59.858821 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848224    4327 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:18:59.858821 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848227    4327 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:18:59.858821 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848230    4327 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:18:59.858821 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848232    4327 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:18:59.858821 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848235    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:18:59.859302 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848237    4327 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:18:59.859302 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848240    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:18:59.859302 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848243    4327 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:18:59.859302 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848246    4327 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:18:59.859302 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848248    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:18:59.859302 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848251    4327 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:18:59.859302 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848253    4327 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:18:59.859302 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848256    4327 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:18:59.859302 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848259    4327 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:18:59.859302 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848262    4327 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:18:59.859302 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848265    4327 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:18:59.859302 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848268    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:18:59.859302 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848271    4327 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:18:59.859302 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848273    4327 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:18:59.859302 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848276    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:18:59.859302 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848279    4327 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:18:59.859302 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848281    4327 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:18:59.859302 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848284    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:18:59.859302 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848286    4327 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:18:59.859764 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848289    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:18:59.859764 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848291    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:18:59.859764 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848294    4327 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:18:59.859764 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848298    4327 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:18:59.859764 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848301    4327 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:18:59.859764 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848304    4327 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:18:59.859764 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848307    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:18:59.859764 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848309    4327 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:18:59.859764 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848312    4327 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:18:59.859764 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848315    4327 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:18:59.859764 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848319    4327 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:18:59.859764 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848322    4327 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:18:59.859764 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848326    4327 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:18:59.859764 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848329    4327 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:18:59.859764 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848332    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:18:59.859764 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848335    4327 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:18:59.859764 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848338    4327 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:18:59.859764 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848340    4327 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:18:59.859764 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848343    4327 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:18:59.860227 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848345    4327 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:18:59.860227 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848348    4327 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:18:59.860227 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848352    4327 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:18:59.860227 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848355    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:18:59.860227 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848357    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:18:59.860227 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848361    4327 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:18:59.860227 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848364    4327 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:18:59.860227 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848366    4327 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:18:59.860227 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848369    4327 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:18:59.860227 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848372    4327 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:18:59.860227 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848374    4327 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:18:59.860227 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848377    4327 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:18:59.860227 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848379    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:18:59.860227 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848382    4327 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:18:59.860227 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848385    4327 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:18:59.860227 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848388    4327 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:18:59.860227 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848390    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:18:59.860227 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848393    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:18:59.860227 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848396    4327 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:18:59.860689 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.848422    4327 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:18:59.860689 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848550    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:18:59.860689 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848554    4327 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:18:59.860689 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848558    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:18:59.860689 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848561    4327 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:18:59.860689 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848564    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:18:59.860689 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848567    4327 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:18:59.860689 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848569    4327 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:18:59.860689 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848572    4327 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:18:59.860689 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848576    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:18:59.860689 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848579    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:18:59.860689 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848581    4327 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:18:59.860689 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848584    4327 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:18:59.860689 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848587    4327 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:18:59.860689 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848590    4327 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:18:59.860689 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848593    4327 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:18:59.861078 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848596    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:18:59.861078 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848598    4327 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:18:59.861078 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848601    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:18:59.861078 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848604    4327 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:18:59.861078 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848607    4327 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:18:59.861078 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848610    4327 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:18:59.861078 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848612    4327 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:18:59.861078 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848615    4327 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:18:59.861078 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848617    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:18:59.861078 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848620    4327 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:18:59.861078 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848623    4327 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:18:59.861078 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848625    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:18:59.861078 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848628    4327 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:18:59.861078 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848631    4327 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:18:59.861078 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848634    4327 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:18:59.861078 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848636    4327 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:18:59.861078 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848639    4327 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:18:59.861078 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848642    4327 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:18:59.861078 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848644    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:18:59.861078 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848647    4327 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:18:59.861572 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848650    4327 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:18:59.861572 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848653    4327 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:18:59.861572 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848655    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:18:59.861572 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848658    4327 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:18:59.861572 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848661    4327 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:18:59.861572 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848663    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:18:59.861572 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848666    4327 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:18:59.861572 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848670    4327 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:18:59.861572 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848673    4327 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:18:59.861572 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848691    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:18:59.861572 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848694    4327 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:18:59.861572 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848697    4327 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:18:59.861572 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848700    4327 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:18:59.861572 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848703    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:18:59.861572 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848706    4327 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:18:59.861572 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848709    4327 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:18:59.861572 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848712    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:18:59.861572 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848715    4327 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:18:59.861572 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848717    4327 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:18:59.861572 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848720    4327 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:18:59.862067 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848723    4327 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:18:59.862067 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848726    4327 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:18:59.862067 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848728    4327 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:18:59.862067 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848731    4327 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:18:59.862067 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848734    4327 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:18:59.862067 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848736    4327 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:18:59.862067 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848739    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:18:59.862067 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848741    4327 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:18:59.862067 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848744    4327 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:18:59.862067 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848747    4327 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:18:59.862067 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848749    4327 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:18:59.862067 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848752    4327 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:18:59.862067 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848754    4327 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:18:59.862067 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848757    4327 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:18:59.862067 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848759    4327 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:18:59.862067 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848762    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:18:59.862067 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848764    4327 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:18:59.862067 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848767    4327 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:18:59.862067 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848770    4327 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:18:59.862519 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848772    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:18:59.862519 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848775    4327 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:18:59.862519 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848777    4327 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:18:59.862519 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848780    4327 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:18:59.862519 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848783    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:18:59.862519 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848785    4327 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:18:59.862519 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848788    4327 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:18:59.862519 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848791    4327 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:18:59.862519 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848794    4327 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:18:59.862519 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848796    4327 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:18:59.862519 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848799    4327 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:18:59.862519 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848802    4327 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:18:59.862519 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848804    4327 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:18:59.862519 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848809    4327 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:18:59.862519 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848812    4327 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:18:59.862519 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848815    4327 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:18:59.862519 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848818    4327 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:18:59.862519 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848821    4327 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:18:59.862519 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848824    4327 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:18:59.862988 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848826    4327 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:18:59.862988 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848829    4327 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:18:59.862988 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848831    4327 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:18:59.862988 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848834    4327 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:18:59.862988 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848837    4327 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:18:59.862988 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848839    4327 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:18:59.862988 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848842    4327 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:18:59.862988 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848844    4327 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:18:59.862988 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848847    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:18:59.862988 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848849    4327 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:18:59.862988 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848852    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:18:59.862988 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848855    4327 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:18:59.862988 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848857    4327 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:18:59.862988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.848880    4327 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:18:59.862988 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848911    4327 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:18:59.862988 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848915    4327 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:18:59.863365 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848918    4327 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:18:59.863365 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848922    4327 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:18:59.863365 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848924    4327 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:18:59.863365 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848927    4327 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:18:59.863365 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848930    4327 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:18:59.863365 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848933    4327 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:18:59.863365 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848935    4327 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:18:59.863365 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848938    4327 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:18:59.863365 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848941    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:18:59.863365 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848943    4327 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:18:59.863365 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848946    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:18:59.863365 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848949    4327 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:18:59.863365 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848952    4327 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:18:59.863365 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848954    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:18:59.863365 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848957    4327 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:18:59.863365 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848960    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:18:59.863365 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848963    4327 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:18:59.863365 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848965    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:18:59.863365 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848968    4327 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:18:59.863365 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848971    4327 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:18:59.863888 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848974    4327 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:18:59.863888 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848976    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:18:59.863888 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848979    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:18:59.863888 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848981    4327 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:18:59.863888 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848984    4327 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:18:59.863888 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848986    4327 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:18:59.863888 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848989    4327 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:18:59.863888 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848992    4327 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:18:59.863888 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848994    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:18:59.863888 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848997    4327 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:18:59.863888 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.848999    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:18:59.863888 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849002    4327 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:18:59.863888 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849004    4327 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:18:59.863888 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849007    4327 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:18:59.863888 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849010    4327 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:18:59.863888 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849012    4327 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:18:59.863888 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849015    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:18:59.863888 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849018    4327 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:18:59.863888 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849021    4327 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:18:59.864331 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849024    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:18:59.864331 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849026    4327 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:18:59.864331 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849029    4327 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:18:59.864331 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849032    4327 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:18:59.864331 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849037    4327 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:18:59.864331 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849040    4327 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:18:59.864331 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849043    4327 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:18:59.864331 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849046    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:18:59.864331 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849049    4327 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:18:59.864331 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849052    4327 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:18:59.864331 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849054    4327 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:18:59.864331 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849057    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:18:59.864331 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849060    4327 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:18:59.864331 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849063    4327 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:18:59.864331 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849065    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:18:59.864331 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849068    4327 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:18:59.864331 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849072    4327 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:18:59.864331 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849075    4327 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:18:59.864331 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849079    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:18:59.864788 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849083    4327 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:18:59.864788 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849086    4327 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:18:59.864788 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849089    4327 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:18:59.864788 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849091    4327 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:18:59.864788 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849094    4327 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:18:59.864788 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849096    4327 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:18:59.864788 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849099    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:18:59.864788 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849102    4327 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:18:59.864788 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849104    4327 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:18:59.864788 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849107    4327 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:18:59.864788 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849109    4327 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:18:59.864788 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849112    4327 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:18:59.864788 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849114    4327 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:18:59.864788 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849118    4327 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:18:59.864788 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849120    4327 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:18:59.864788 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849123    4327 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:18:59.864788 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849126    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:18:59.864788 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849129    4327 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:18:59.864788 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849132    4327 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:18:59.864788 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849135    4327 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:18:59.865292 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849137    4327 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:18:59.865292 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849140    4327 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:18:59.865292 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849143    4327 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:18:59.865292 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849145    4327 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:18:59.865292 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849148    4327 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:18:59.865292 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849151    4327 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:18:59.865292 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849153    4327 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:18:59.865292 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849156    4327 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:18:59.865292 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849159    4327 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:18:59.865292 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849161    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:18:59.865292 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849164    4327 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:18:59.865292 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849167    4327 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:18:59.865292 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849171    4327 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:18:59.865292 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849174    4327 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:18:59.865292 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849177    4327 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:18:59.865292 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849180    4327 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:18:59.865292 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849183    4327 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:18:59.865292 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849185    4327 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:18:59.865292 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849188    4327 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:18:59.865292 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849190    4327 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:18:59.865901 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849193    4327 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:18:59.865901 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849195    4327 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:18:59.865901 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849200    4327 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:18:59.865901 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849204    4327 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:18:59.865901 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849208    4327 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:18:59.865901 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:18:59.849211    4327 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:18:59.865901 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.849257    4327 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 00:18:59.865901 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.850218    4327 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 00:18:59.865901 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.850270    4327 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 00:18:59.865901 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.850448    4327 server.go:1014] "Starting client certificate rotation"
Oct 31 00:18:59.865901 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.850603    4327 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 00:18:59.865901 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.850669    4327 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:13:19 +0000 UTC" deadline="2028-07-10 23:02:08.693291916 +0000 UTC"
Oct 31 00:18:59.865901 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.850731    4327 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14854h43m8.842565059s"
Oct 31 00:18:59.865901 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.850907    4327 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 00:18:59.865901 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.851903    4327 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 00:18:59.866265 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.853452    4327 log.go:25] "Validated CRI v1 runtime API"
Oct 31 00:18:59.866265 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.855062    4327 log.go:25] "Validated CRI v1 image API"
Oct 31 00:18:59.866265 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.856190    4327 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 00:18:59.866265 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.858077    4327 fs.go:135] Filesystem UUIDs: map[778d9087-c106-4487-91b0-5334878adcc3:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 88d3d325-1b23-43d0-b4d6-8af4f06ba1cf:/dev/nvme0n1p4]
Oct 31 00:18:59.866383 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.858099    4327 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/033e943c5f2a00f0e0cd21ab0644ebef297bab6faa89eb64c4b9364c6d1900e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/033e943c5f2a00f0e0cd21ab0644ebef297bab6faa89eb64c4b9364c6d1900e3/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0bde3ead4ffd0b3a9857b98bf246d4d6ba0a513dadbbe5dbf7b13319160da364/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0bde3ead4ffd0b3a9857b98bf246d4d6ba0a513dadbbe5dbf7b13319160da364/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/11ce4c9c645862b529ca4e56f87a3c0335b7172e5515b8ab3b93d995e3942425/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/11ce4c9c645862b529ca4e56f87a3c0335b7172e5515b8ab3b93d995e3942425/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/413933e427d61a4d7788dd590ddd013dd2af5d6a1d89cad636cca4365cc875b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/413933e427d61a4d7788dd590ddd013dd2af5d6a1d89cad636cca4365cc875b8/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5baf3e2893288f4a3179df9ca4738c5072442391df2aea0617d11ba58ce57975/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5baf3e2893288f4a3179df9ca4738c5072442391df2aea0617d11ba58ce57975/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c8dc8b6dcc79627f55cf1e5bff9ad8dc3e3ed9eb7e1cf8f4ad9d88aecd707eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c8dc8b6dcc79627f55cf1e5bff9ad8dc3e3ed9eb7e1cf8f4ad9d88aecd707eb/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/93de93df25f639d3b26ba4c93369d11af58424ab46e708468f2273c84b6b7cb8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/93de93df25f639d3b26ba4c93369d11af58424ab46e708468f2273c84b6b7cb8/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5a25be85fe813b43a6e7973d872017c0d5b8552f078b3a1c0eac4df30054064/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5a25be85fe813b43a6e7973d872017c0d5b8552f078b3a1c0eac4df30054064/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b73bdbdfbf94a552b042bbcfb5d528e3de51d3cf0c5f5b7ec764f58971cdc684/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b73bdbdfbf94a552b042bbcfb5d528e3de51d3cf0c5f5b7ec764f58971cdc684/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c20b6ee53b5607496b962c020108ac6e9c8847d050b4c924742840c01fb84bed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c20b6ee53b5607496b962c020108ac6e9c8847d050b4c924742840c01fb84bed/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c512452c6fa7183057ed9b9881a52da2709559b7b82122786b548786479c4c29/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c512452c6fa7183057ed9b9881a52da2709559b7b82122786b548786479c4c29/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc3c27ce2cac021f57f143020d30cb0b82170126b74b7091267f0c5746542114/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc3c27ce2cac021f57f143020d30cb0b82170126b74b7091267f0c5746542114/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf16bf06fb01a647c3bfe579684dc0d24ef6a9de7c246b2befb30a90bfd3fe58/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf16bf06fb01a647c3bfe579684dc0d24ef6a9de7c246b2befb30a90bfd3fe58/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dfa1b91dce127f457caac2591af59bec0d2ea3b18c56ff5080cabe0045f3e3ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dfa1b91dce127f457caac2591af59bec0d2ea3b18c56ff5080cabe0045f3e3ba/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0b3b13ccfd639cd31f61fb38d6c5c9d4434b54ca45e016d58b8929ac0384cc0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0b3b13ccfd639cd31f61fb38d6c5c9d4434b54ca45e016d58b8929ac0384cc0/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/002d2da2-bb18-4667-98c8-1770b70d9985/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/002d2da2-bb18-4667-98c8-1770b70d9985/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49/volumes/kubernetes.io~projected/kube-api-access-mkmrt:{mountpoint:/var/lib/kubelet/pods/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49/volumes/kubernetes.io~projected/kube-api-access-mkmrt major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17401e80-fe41-417f-9840-acd7a68d17e3/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/17401e80-fe41-417f-9840-acd7a68d17e3/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f456866-ce05-42c8-adb6-588cf5fcd8b7/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/1f456866-ce05-42c8-adb6-588cf5fcd8b7/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f456866-ce05-42c8-adb6-588cf5fcd8b7/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/1f456866-ce05-42c8-adb6-588cf5fcd8b7/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f456866-ce05-42c8-adb6-588cf5fcd8b7/volumes/kubernetes.io~projected/kube-api-access-755hv:{mountpoint:/var/lib/kubelet/pods/1f456866-ce05-42c8-adb6-588cf5fcd8b7/volumes/kubernetes.io~projected/kube-api-access-755hv major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a/volumes/kubernetes.io~projected/kube-api-access-7bwkn:{mountpoint:/var/lib/kubelet/pods/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a/volumes/kubernetes.io~projected/kube-api-access-7bwkn major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fdd0807-bb6c-4867-aad0-40af2ddc7d08/volumes/kubernetes.io~projected/kube-api-access-6bbs6:{mountpoint:/var/lib/kubelet/pods/4fdd0807-bb6c-4867-aad0-40af2ddc7d08/volumes/kubernetes.io~projected/kube-api-access-6bbs6 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b9f4a03-68eb-418e-9447-ad62473a1e9a/volumes/kubernetes.io~projected/kube-api-access-ngkbc:{mountpoint:/var/lib/kubelet/pods/5b9f4a03-68eb-418e-9447-ad62473a1e9a/volumes/kubernetes.io~projected/kube-api-access-ngkbc major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79ab34ed-5163-4cff-8cf4-170c3445e655/volumes/kubernetes.io~projected/kube-api-access-jvxxw:{mountpoint:/var/lib/kubelet/pods/79ab34ed-5163-4cff-8cf4-170c3445e655/volumes/kubernetes.io~projected/kube-api-access-jvxxw major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79ab34ed-5163-4cff-8cf4-170c3445e655/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/79ab34ed-5163-4cff-8cf4-170c3445e655/volumes/kubernetes.io~secret/metrics-tls major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8072caf6-1730-4100-a87a-630ace22a370/volumes/kubernetes.io~projected/kube-api-access-q65jt:{mountpoint:/var/lib/kubelet/pods/8072caf6-1730-4100-a87a-630ace22a370/volumes/kubernetes.io~projected/kube-api-access-q65jt major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8072caf6-1730-4100-a87a-630ace22a370/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/8072caf6-1730-4100-a87a-630ace22a370/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84a754a9-1c06-4302-bb01-4f53a1e032d0/volumes/kubernetes.io~projected/kube-api-access-fwvkk:{mountpoint:/var/lib/kubelet/pods/84a754a9-1c06-4302-bb01-4f53a1e032d0/volumes/kubernetes.io~projected/kube-api-access-fwvkk major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95f3be30-2d65-4373-9623-155c841dca11/volumes/kubernetes.io~projected/kube-api-access-mndzv:{mountpoint:/var/lib/kubelet/pods/95f3be30-2d65-4373-9623-155c841dca11/volumes/kubernetes.io~projected/kube-api-access-mndzv major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8190e88-1887-415f-95cb-d421f3fbc08b/volumes/kubernetes.io~projected/kube-api-access-fk4vp:{mountpoint:/var/lib/kubelet/pods/a8190e88-1887-415f-95cb-d421f3fbc08b/volumes/kubernetes.io~projected/kube-api-access-fk4vp major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0c02c18-5891-493e-ab22-5191677b8364/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/b0c02c18-5891-493e-ab22-5191677b8364/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0c02c18-5891-493e-ab22-5191677b8364/volumes/kubernetes.io~projected/kube-api-access-88p6w:{mountpoint:/var/lib/kubelet/pods/b0c02c18-5891-493e-ab22-5191677b8364/volumes/kubernetes.io~projected/kube-api-access-88p6w major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0c02c18-5891-493e-ab22-5191677b8364/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/b0c02c18-5891-493e-ab22-5191677b8364/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbc0fbb5-0386-40b9-aeef-3b7aa867f625/volumes/kubernetes.io~projected/kube-api-access-m7pbg:{mountpoint:/var/lib/kubelet/pods/bbc0fbb5-0386-40b9-aeef-3b7aa867f625/volumes/kubernetes.io~projected/kube-api-access-m7pbg major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec6a6238-81be-4a82-b346-b4c9fe3aefa4/volumes/kubernetes.io~projected/kube-api-access-rz8l7:{mountpoint:/var/lib/kubelet/pods/ec6a6238-81be-4a82-b346-b4c9fe3aefa4/volumes/kubernetes.io~projected/kube-api-access-rz8l7 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f18d4e41-daf0-4be5-bcdb-c00d189419ee/volumes/kubernetes.io~projected/kube-api-access-bvblv:{mountpoint:/var/lib/kubelet/pods/f18d4e41-daf0-4be5-bcdb-c00d189419ee/volumes/kubernetes.io~projected/kube-api-access-bvblv major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f18d4e41-daf0-4be5-bcdb-c00d189419ee/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/f18d4e41-daf0-4be5-bcdb-c00d189419ee/volumes/kubernetes.io~secret/signing-key major:0 minor:203 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/38f7c042453f4f4fba3dc571e5f9d5141b8e6f79a0648fc60ed3be1894eade41/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/694d4b9f380052375aa4508abfdb77af064b468b59a65e03084a918912098ae4/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/d38f5900ecebe5899686e409e59e93f64e542ff9f257302d526fcc33f7fdbbfb/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/2ff9dead73236e9a14277845cd181f7d03e06108a5442a7580b33492fbc7b372/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/5db53262c567441f02983e199bbfcb19db9a3992c20cef8dd1b41ef68e070713/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/b6a38006f0e5394811f95f9352de9f5f352d13952e998dda358d265ef5416987/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/71ba9781e584d0834cb564ec8efb1da37acf5a9d86c832ee537afed5bfb04c96/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/a28408480b217733445b9152d4d358452b4156a94cf7521d7d434379e2eeeb92/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/d2c97fceaa3591f21e096561b882b005743f41d170554a028d5a8c26b4715bdc/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/3dc944a6f03897a442ca69f752102d33b69a5246839037900c9f69b76f944863/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/c270f360651b18be8d93819f69026acc60049022f4725aa01f0fc1dc268a21d6/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/6505683937b46128f4d16b5514fcc51eb3910347bb6b8e7d33ecf6a4e6580e5b/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/b81e82897e84cb3d409d151739cff11c38f244f92274584604e4421874917509/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/7511de48fcff035d124d6a379bcc8fec6d2d4f343d89bd80f05c8746c9c13a88/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/e719d083c1319bd0432fbc14b12355d74145d2414233d34c8fa9ba289bb774b4/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/d3b645529a8a72cf7bbee2452d8e40d6c2d3e5d259fdc1228cd00f1d7ce258d6/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/71cdd07dfa9b1991c583ace34dc3e7bb6cb30db94d08d1cc8d0e31a6556b2463/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/a56542f50e77d1ca031ce9acf11e4aa547d8400a6339904816ce10afe540577b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/c0c7a60fe54bb54f349e8443f3eb597721c5b7e1d39b2904920c6e3339f376eb/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/11791c5bc0cf87d902e651434a27f1549220e6566e70bf03c637d4b48b4c6409/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/7c2b3b6a2810d34bfdc4c076c3b049112a7096956d1adadec86c70d459feb595/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/32e75ea4016e6b35030a9df6883d6285c6518b6c084b3a531fbd84c1a5e78b11/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/5973fcd638c4f740ea793e4cc76b88cab3c60c796e442c80837afddebb8c2c2c/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/ec7a67317053e2df8938bc6a104a1e00cc4f2d00569cca850daef42477ae2ea1/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/94d11f0af5bd4e8a9c4103c9455ba68c4294cca225c8fa1aaeaa8a12b626f0d6/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/3be1342fce2b0b24358055c03ee9a211634dfd69f09ceb8055239d1c5847efa9/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/658e83215306df3ffcf0b330b2b04243e68399ddb2c33fa2d391235adebd5cf0/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-238:{mountpoint:/var/lib/containers/storage/overlay/968757e25f3a11392eadf3752b6a1a167b17b1ea9a83e6f589afec1a7357c93e/merged major:0 minor:238 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/25e1267853671ed9a041399f1ae67c8e5ef8e914e7355c2e7c8a04ba1eaa5097/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/e72c562f8b32af5aeb97c94366f6698855c62b45c0430b76704b910c150d75f1/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/289ba8f4e2e7e25632111284322d1713d0a7ad55a131cb06d5ccf24955439dbc/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/937471c9a306359705d85b80457d7e146a221c768bc1daf68f2ad5a2abbda8bd/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/76fa6805ea3b136762dad3be69a5d3fb3aad7598cccfff0b1ed841eb6562201b/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/43f2331dda26b1957306ee2075a3b5fa298bb87a5c5b3775bee8e2fc2739fceb/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/b343555e92e6ea2472b877cace738ab4058da3fbe833ba368f01518808a276df/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/25df56c6153af905b18083961d2773c54cac122d22d259d3bcd56c51d0186e73/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/adf28e11e97ad9f70c828b5fbc92a1e86075914ff2ad0d355eee8146395f2ac6/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/ec96d17a13d6421fcec4f15581f82ed12f9f6cc1e281bc83751910b2ffbcd508/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/2fe221d47045f9017fd40a83909d54ee1fbfd00389a1858ef26dd43a35839aef/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/25f83189496960656a5d2875f0f994222a0b599a8c88db56132abd69e512ac93/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/3e184c3695bf35773814c36c77660d95b614335de9b95fbc6434c954eeebeae1/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/072065cd34e018be452232a8b3edaf787f61760ab5e65dd74ac10959e57acbd8/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 31 00:18:59.866607 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.864895    4327 manager.go:217] Machine: {Timestamp:2026-10-31 00:18:59.864412032 +0000 UTC m=+0.086447571 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3110515 MemoryCapacity:33164345344 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec257c5ed16ec91d2717d233f5153df8 SystemUUID:ec257c5e-d16e-c91d-2717-d233f5153df8 BootID:beefdae8-801c-4b93-b9a9-7786b8dd848a Filesystems:[{Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5a25be85fe813b43a6e7973d872017c0d5b8552f078b3a1c0eac4df30054064/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0c02c18-5891-493e-ab22-5191677b8364/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b9f4a03-68eb-418e-9447-ad62473a1e9a/volumes/kubernetes.io~projected/kube-api-access-ngkbc DeviceMajor:0 DeviceMinor:226 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/a8190e88-1887-415f-95cb-d421f3fbc08b/volumes/kubernetes.io~projected/kube-api-access-fk4vp DeviceMajor:0 DeviceMinor:63 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8072caf6-1730-4100-a87a-630ace22a370/volumes/kubernetes.io~projected/kube-api-access-q65jt DeviceMajor:0 DeviceMinor:197 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/95f3be30-2d65-4373-9623-155c841dca11/volumes/kubernetes.io~projected/kube-api-access-mndzv DeviceMajor:0 DeviceMinor:59 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79ab34ed-5163-4cff-8cf4-170c3445e655/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:235 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/ec6a6238-81be-4a82-b346-b4c9fe3aefa4/volumes/kubernetes.io~projected/kube-api-access-rz8l7 DeviceMajor:0 DeviceMinor:60 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8072caf6-1730-4100-a87a-630ace22a370/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/f18d4e41-daf0-4be5-bcdb-c00d189419ee/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:203 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582172672 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b73bdbdfbf94a552b042bbcfb5d528e3de51d3cf0c5f5b7ec764f58971cdc684/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0b3b13ccfd639cd31f61fb38d6c5c9d4434b54ca45e016d58b8929ac0384cc0/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/033e943c5f2a00f0e0cd21ab0644ebef297bab6faa89eb64c4b9364c6d1900e3/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/4fdd0807-bb6c-4867-aad0-40af2ddc7d08/volumes/kubernetes.io~projected/kube-api-access-6bbs6 DeviceMajor:0 DeviceMinor:61 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/b0c02c18-5891-493e-ab22-5191677b8364/volumes/kubernetes.io~projected/kube-api-access-88p6w DeviceMajor:0 DeviceMinor:64 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/bbc0fbb5-0386-40b9-aeef-3b7aa867f625/volumes/kubernetes.io~projected/kube-api-access-m7pbg DeviceMajor:0 DeviceMinor:65 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dfa1b91dce127f457caac2591af59bec0d2ea3b18c56ff5080cabe0045f3e3ba/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79ab34ed-5163-4cff-8cf4-170c3445e655/volumes/kubernetes.io~projected/kube-api-access-jvxxw DeviceMajor:0 DeviceMinor:201 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/f18d4e41-daf0-4be5-bcdb-c00d189419ee/volumes/kubernetes.io~projected/kube-api-access-bvblv DeviceMajor:0 DeviceMinor:204 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/11ce4c9c645862b529ca4e56f87a3c0335b7172e5515b8ab3b93d995e3942425/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/413933e427d61a4d7788dd590ddd013dd2af5d6a1d89cad636cca4365cc875b8/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0bde3ead4ffd0b3a9857b98bf246d4d6ba0a513dadbbe5dbf7b13319160da364/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0c02c18-5891-493e-ab22-5191677b8364/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/002d2da2-bb18-4667-98c8-1770b70d9985/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:221 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-238 DeviceMajor:0 DeviceMinor:238 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/93de93df25f639d3b26ba4c93369d11af58424ab46e708468f2273c84b6b7cb8/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5baf3e2893288f4a3179df9ca4738c5072442391df2aea0617d11ba58ce57975/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f456866-ce05-42c8-adb6-588cf5fcd8b7/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/1f456866-ce05-42c8-adb6-588cf5fcd8b7/volumes/kubernetes.io~projected/kube-api-access-755hv DeviceMajor:0 DeviceMinor:62 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c20b6ee53b5607496b962c020108ac6e9c8847d050b4c924742840c01fb84bed/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f456866-ce05-42c8-adb6-588cf5fcd8b7/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49/volumes/kubernetes.io~projected/kube-api-access-mkmrt DeviceMajor:0 DeviceMinor:67 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c512452c6fa7183057ed9b9881a52da2709559b7b82122786b548786479c4c29/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17401e80-fe41-417f-9840-acd7a68d17e3/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc3c27ce2cac021f57f143020d30cb0b82170126b74b7091267f0c5746542114/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c8dc8b6dcc79627f55cf1e5bff9ad8dc3e3ed9eb7e1cf8f4ad9d88aecd707eb/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a/volumes/kubernetes.io~projected/kube-api-access-7bwkn DeviceMajor:0 DeviceMinor:66 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf16bf06fb01a647c3bfe579684dc0d24ef6a9de7c246b2befb30a90bfd3fe58/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/84a754a9-1c06-4302-bb01-4f53a1e032d0/volumes/kubernetes.io~projected/kube-api-access-fwvkk DeviceMajor:0 DeviceMinor:202 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:033e943c5f2a00f MacAddress:7a:af:e7:3f:19:cf Speed:10000 Mtu:8901} {Name:413933e427d61a4 MacAddress:f6:e6:98:f4:0a:e8 Speed:10000 Mtu:8901} {Name:93de93df25f639d MacAddress:4a:62:fe:f8:22:49 Speed:10000 Mtu:8901} {Name:a5a25be85fe813b MacAddress:26:bf:d7:5d:bc:99 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ed:bb:92:73 Speed:0 Mtu:9001} {Name:br-int MacAddress:a2:43:3f:ac:25:51 Speed:0 Mtu:8901} {Name:cc3c27ce2cac021 MacAddress:c6:91:24:ba:d8:14 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ed:bb:92:73 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:a2:50:66:f2:57:9a Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:e6:f0:76:2d:93:ef Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164345344 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 00:18:59.866607 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.865262    4327 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 00:18:59.866607 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.865357    4327 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 00:18:59.866607 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.865644    4327 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 00:18:59.866607 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.865664    4327 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-253.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 00:18:59.866607 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.865827    4327 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 00:18:59.866607 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.865835    4327 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 00:18:59.866607 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.865848    4327 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 00:18:59.866607 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.865866    4327 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 00:18:59.866607 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.865936    4327 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 00:18:59.866607 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.866074    4327 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 00:18:59.866607 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.866325    4327 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 00:18:59.866607 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.866360    4327 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 00:18:59.866607 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.866369    4327 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 00:18:59.866607 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.866378    4327 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 00:18:59.866607 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.866387    4327 kubelet.go:405] "Adding apiserver pod source"
Oct 31 00:18:59.866607 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.866402    4327 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 00:18:59.867287 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.866924    4327 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 00:18:59.867287 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.866936    4327 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 00:18:59.867447 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.867386    4327 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 00:18:59.869201 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.869189    4327 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 00:18:59.869201 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.869201    4327 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 00:18:59.869304 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.869295    4327 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 00:18:59.869485 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.869474    4327 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 00:18:59.869522 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.869504    4327 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 00:18:59.869602 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.869593    4327 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 00:18:59.869635 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.869608    4327 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 00:18:59.869635 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.869616    4327 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 00:18:59.869635 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.869621    4327 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 00:18:59.869635 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.869627    4327 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 00:18:59.869635 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.869633    4327 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 00:18:59.869795 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.869643    4327 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 00:18:59.869795 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.869651    4327 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 00:18:59.869795 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.869658    4327 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 00:18:59.869795 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.869664    4327 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 00:18:59.869795 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.869699    4327 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 00:18:59.869795 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.869709    4327 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 00:18:59.869795 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.869736    4327 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 00:18:59.869795 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.869744    4327 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 00:18:59.873034 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.873020    4327 server.go:1263] "Started kubelet"
Oct 31 00:18:59.873118 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.873074    4327 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 00:18:59.873641 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.873588    4327 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 00:18:59.873782 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.873697    4327 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 00:18:59.873657 ip-10-0-128-253 systemd[1]: Started Kubernetes Kubelet.
Oct 31 00:18:59.874127 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.874082    4327 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 00:18:59.876173 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.876154    4327 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 00:18:59.878631 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.878611    4327 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 00:18:59.878631 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.878631    4327 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 00:18:59.879102 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.878695    4327 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:13:21 +0000 UTC" deadline="2028-03-26 04:17:24.589220383 +0000 UTC"
Oct 31 00:18:59.879102 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.878713    4327 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12291h58m24.710510461s"
Oct 31 00:18:59.879102 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.878741    4327 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 00:18:59.879102 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.878756    4327 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 00:18:59.879102 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:18:59.878762    4327 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-253.ec2.internal\" not found"
Oct 31 00:18:59.879102 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.878964    4327 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 00:18:59.880881 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.880844    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79ab34ed-5163-4cff-8cf4-170c3445e655" volumeName="kubernetes.io/configmap/79ab34ed-5163-4cff-8cf4-170c3445e655-config-volume" seLinuxMountContext=""
Oct 31 00:18:59.880978 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.880885    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79ab34ed-5163-4cff-8cf4-170c3445e655" volumeName="kubernetes.io/projected/79ab34ed-5163-4cff-8cf4-170c3445e655-kube-api-access-jvxxw" seLinuxMountContext=""
Oct 31 00:18:59.880978 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.880900    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0c02c18-5891-493e-ab22-5191677b8364" volumeName="kubernetes.io/configmap/b0c02c18-5891-493e-ab22-5191677b8364-env-overrides" seLinuxMountContext=""
Oct 31 00:18:59.880978 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.880911    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec6a6238-81be-4a82-b346-b4c9fe3aefa4" volumeName="kubernetes.io/projected/ec6a6238-81be-4a82-b346-b4c9fe3aefa4-kube-api-access-rz8l7" seLinuxMountContext=""
Oct 31 00:18:59.880978 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.880923    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0c02c18-5891-493e-ab22-5191677b8364" volumeName="kubernetes.io/configmap/b0c02c18-5891-493e-ab22-5191677b8364-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 00:18:59.880978 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.880934    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f456866-ce05-42c8-adb6-588cf5fcd8b7" volumeName="kubernetes.io/projected/1f456866-ce05-42c8-adb6-588cf5fcd8b7-kube-api-access-755hv" seLinuxMountContext=""
Oct 31 00:18:59.880978 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.880946    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fdd0807-bb6c-4867-aad0-40af2ddc7d08" volumeName="kubernetes.io/configmap/4fdd0807-bb6c-4867-aad0-40af2ddc7d08-iptables-alerter-script" seLinuxMountContext=""
Oct 31 00:18:59.880978 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.880957    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b9f4a03-68eb-418e-9447-ad62473a1e9a" volumeName="kubernetes.io/projected/5b9f4a03-68eb-418e-9447-ad62473a1e9a-kube-api-access-ngkbc" seLinuxMountContext=""
Oct 31 00:18:59.880978 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.880972    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8072caf6-1730-4100-a87a-630ace22a370" volumeName="kubernetes.io/projected/8072caf6-1730-4100-a87a-630ace22a370-kube-api-access-q65jt" seLinuxMountContext=""
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.880983    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8190e88-1887-415f-95cb-d421f3fbc08b" volumeName="kubernetes.io/configmap/a8190e88-1887-415f-95cb-d421f3fbc08b-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.880994    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbc0fbb5-0386-40b9-aeef-3b7aa867f625" volumeName="kubernetes.io/configmap/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-cni-binary-copy" seLinuxMountContext=""
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881005    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f456866-ce05-42c8-adb6-588cf5fcd8b7" volumeName="kubernetes.io/empty-dir/1f456866-ce05-42c8-adb6-588cf5fcd8b7-tmp" seLinuxMountContext=""
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881016    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79ab34ed-5163-4cff-8cf4-170c3445e655" volumeName="kubernetes.io/empty-dir/79ab34ed-5163-4cff-8cf4-170c3445e655-tmp-dir" seLinuxMountContext=""
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881027    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f18d4e41-daf0-4be5-bcdb-c00d189419ee" volumeName="kubernetes.io/secret/f18d4e41-daf0-4be5-bcdb-c00d189419ee-signing-key" seLinuxMountContext=""
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881043    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a" volumeName="kubernetes.io/projected/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-kube-api-access-7bwkn" seLinuxMountContext=""
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881063    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79ab34ed-5163-4cff-8cf4-170c3445e655" volumeName="kubernetes.io/secret/79ab34ed-5163-4cff-8cf4-170c3445e655-metrics-tls" seLinuxMountContext=""
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881065    4327 factory.go:55] Registering systemd factory
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881074    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8072caf6-1730-4100-a87a-630ace22a370" volumeName="kubernetes.io/secret/8072caf6-1730-4100-a87a-630ace22a370-cert" seLinuxMountContext=""
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881086    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84a754a9-1c06-4302-bb01-4f53a1e032d0" volumeName="kubernetes.io/configmap/84a754a9-1c06-4302-bb01-4f53a1e032d0-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881098    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84a754a9-1c06-4302-bb01-4f53a1e032d0" volumeName="kubernetes.io/projected/84a754a9-1c06-4302-bb01-4f53a1e032d0-kube-api-access-fwvkk" seLinuxMountContext=""
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881109    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0c02c18-5891-493e-ab22-5191677b8364" volumeName="kubernetes.io/projected/b0c02c18-5891-493e-ab22-5191677b8364-kube-api-access-88p6w" seLinuxMountContext=""
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881118    4327 factory.go:223] Registration of the systemd container factory successfully
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881125    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="165b1a7a-d8ce-49d4-9a08-95a3cbea4f49" volumeName="kubernetes.io/projected/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-kube-api-access-mkmrt" seLinuxMountContext=""
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881147    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17401e80-fe41-417f-9840-acd7a68d17e3" volumeName="kubernetes.io/secret/17401e80-fe41-417f-9840-acd7a68d17e3-agent-certs" seLinuxMountContext=""
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881174    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8190e88-1887-415f-95cb-d421f3fbc08b" volumeName="kubernetes.io/configmap/a8190e88-1887-415f-95cb-d421f3fbc08b-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881189    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8190e88-1887-415f-95cb-d421f3fbc08b" volumeName="kubernetes.io/projected/a8190e88-1887-415f-95cb-d421f3fbc08b-kube-api-access-fk4vp" seLinuxMountContext=""
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881201    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec6a6238-81be-4a82-b346-b4c9fe3aefa4" volumeName="kubernetes.io/empty-dir/ec6a6238-81be-4a82-b346-b4c9fe3aefa4-tmp-dir" seLinuxMountContext=""
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881213    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f18d4e41-daf0-4be5-bcdb-c00d189419ee" volumeName="kubernetes.io/configmap/f18d4e41-daf0-4be5-bcdb-c00d189419ee-signing-cabundle" seLinuxMountContext=""
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881226    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17401e80-fe41-417f-9840-acd7a68d17e3" volumeName="kubernetes.io/configmap/17401e80-fe41-417f-9840-acd7a68d17e3-konnectivity-ca" seLinuxMountContext=""
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881246    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fdd0807-bb6c-4867-aad0-40af2ddc7d08" volumeName="kubernetes.io/projected/4fdd0807-bb6c-4867-aad0-40af2ddc7d08-kube-api-access-6bbs6" seLinuxMountContext=""
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881264    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8190e88-1887-415f-95cb-d421f3fbc08b" volumeName="kubernetes.io/configmap/a8190e88-1887-415f-95cb-d421f3fbc08b-cni-binary-copy" seLinuxMountContext=""
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881276    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0c02c18-5891-493e-ab22-5191677b8364" volumeName="kubernetes.io/secret/b0c02c18-5891-493e-ab22-5191677b8364-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881289    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbc0fbb5-0386-40b9-aeef-3b7aa867f625" volumeName="kubernetes.io/configmap/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-multus-daemon-config" seLinuxMountContext=""
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881300    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbc0fbb5-0386-40b9-aeef-3b7aa867f625" volumeName="kubernetes.io/projected/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-kube-api-access-m7pbg" seLinuxMountContext=""
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881328    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95f3be30-2d65-4373-9623-155c841dca11" volumeName="kubernetes.io/projected/95f3be30-2d65-4373-9623-155c841dca11-kube-api-access-mndzv" seLinuxMountContext=""
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881341    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="002d2da2-bb18-4667-98c8-1770b70d9985" volumeName="kubernetes.io/secret/002d2da2-bb18-4667-98c8-1770b70d9985-original-pull-secret" seLinuxMountContext=""
Oct 31 00:18:59.881338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881353    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f456866-ce05-42c8-adb6-588cf5fcd8b7" volumeName="kubernetes.io/empty-dir/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-tuned" seLinuxMountContext=""
Oct 31 00:18:59.882623 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881365    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84a754a9-1c06-4302-bb01-4f53a1e032d0" volumeName="kubernetes.io/empty-dir/84a754a9-1c06-4302-bb01-4f53a1e032d0-data-volume" seLinuxMountContext=""
Oct 31 00:18:59.882623 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881378    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95f3be30-2d65-4373-9623-155c841dca11" volumeName="kubernetes.io/configmap/95f3be30-2d65-4373-9623-155c841dca11-serviceca" seLinuxMountContext=""
Oct 31 00:18:59.882623 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881403    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0c02c18-5891-493e-ab22-5191677b8364" volumeName="kubernetes.io/configmap/b0c02c18-5891-493e-ab22-5191677b8364-ovnkube-config" seLinuxMountContext=""
Oct 31 00:18:59.882623 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881415    4327 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f18d4e41-daf0-4be5-bcdb-c00d189419ee" volumeName="kubernetes.io/projected/f18d4e41-daf0-4be5-bcdb-c00d189419ee-kube-api-access-bvblv" seLinuxMountContext=""
Oct 31 00:18:59.882623 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881427    4327 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 00:18:59.882623 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.881435    4327 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 00:18:59.882931 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.882816    4327 factory.go:153] Registering CRI-O factory
Oct 31 00:18:59.882931 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.882833    4327 factory.go:223] Registration of the crio container factory successfully
Oct 31 00:18:59.883023 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.882948    4327 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 00:18:59.883273 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.882969    4327 factory.go:103] Registering Raw factory
Oct 31 00:18:59.883340 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.883278    4327 manager.go:1199] Started watching for new ooms in manager
Oct 31 00:18:59.883566 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.883553    4327 manager.go:319] Starting recovery of all containers
Oct 31 00:18:59.910935 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.910904    4327 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 00:18:59.912536 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.912515    4327 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 00:18:59.912536 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.912538    4327 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 00:18:59.912966 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.912952    4327 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 00:18:59.913045 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:18:59.913001    4327 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 00:18:59.936317 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.936297    4327 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 00:18:59.939555 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.939525    4327 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:18:59.939757 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.939670    4327 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:18:59.943327 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.943300    4327 generic.go:356] "Generic (PLEG): container finished" podID="a8190e88-1887-415f-95cb-d421f3fbc08b" containerID="fa650fa93fd7589469468396212d259afe700a261c7abbc683602ae82170cf20" exitCode=0
Oct 31 00:18:59.943327 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.943326    4327 generic.go:356] "Generic (PLEG): container finished" podID="a8190e88-1887-415f-95cb-d421f3fbc08b" containerID="bcd17679cef98b9db708087c03cb9af8e7d54f0db7a099e92bc917f106262185" exitCode=0
Oct 31 00:18:59.943460 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.943334    4327 generic.go:356] "Generic (PLEG): container finished" podID="a8190e88-1887-415f-95cb-d421f3fbc08b" containerID="547fccdc6942c91f27791a17df1847c3298813d349c1b82b9622c7803728be0c" exitCode=0
Oct 31 00:18:59.943460 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.943340    4327 generic.go:356] "Generic (PLEG): container finished" podID="a8190e88-1887-415f-95cb-d421f3fbc08b" containerID="0981b2fe61bcf2135ad2bc877c45a99fabeadb8d62090596889d47d50025f45c" exitCode=0
Oct 31 00:18:59.943460 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.943345    4327 generic.go:356] "Generic (PLEG): container finished" podID="a8190e88-1887-415f-95cb-d421f3fbc08b" containerID="af80054aab992f1e709d5820d15617f7426022967f4b9d39b820e92aff281c7f" exitCode=0
Oct 31 00:18:59.943460 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.943351    4327 generic.go:356] "Generic (PLEG): container finished" podID="a8190e88-1887-415f-95cb-d421f3fbc08b" containerID="02b1aa7b5186e1f998028e17a26cef03929486e1d654f0a2b2ca0bd4bb242815" exitCode=0
Oct 31 00:18:59.946884 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.946859    4327 generic.go:356] "Generic (PLEG): container finished" podID="c35a4f872afa19338c317775c3e775c3" containerID="8b59e0cb2290f46f406233a8a8b14a37f3bba28af7d00c23a39b289387ccfa13" exitCode=0
Oct 31 00:18:59.953927 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.953905    4327 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:18:59.953927 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.953926    4327 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:18:59.954171 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.954157    4327 manager.go:324] Recovery completed
Oct 31 00:18:59.968324 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.968303    4327 cpu_manager.go:226] "Starting" policy="none"
Oct 31 00:18:59.968324 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.968317    4327 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 00:18:59.968456 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.968334    4327 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 00:18:59.968503 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.968488    4327 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 00:18:59.968536 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.968503    4327 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 00:18:59.968536 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.968519    4327 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 00:18:59.968536 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.968524    4327 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 00:18:59.968536 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.968529    4327 policy_none.go:50] "Start"
Oct 31 00:18:59.968631 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.968538    4327 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 00:18:59.968631 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.968546    4327 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 00:18:59.977775 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.977762    4327 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 00:18:59.977829 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.977776    4327 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 00:18:59.977829 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.977783    4327 policy_none.go:44] "Start"
Oct 31 00:18:59.983593 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.983563    4327 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 00:18:59.987120 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.987107    4327 manager.go:341] "Starting Device Plugin manager"
Oct 31 00:18:59.987650 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:18:59.987164    4327 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 00:18:59.987650 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.987651    4327 server.go:87] "Starting device plugin registration server"
Oct 31 00:18:59.987924 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.987907    4327 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 00:18:59.987986 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.987921    4327 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 00:18:59.988039 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.988009    4327 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 00:18:59.988140 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.988100    4327 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 00:18:59.988140 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.988142    4327 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 00:18:59.988293 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.988149    4327 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 00:18:59.988293 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.988188    4327 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:18:59.98813149Z","UUID":"b37ef8b0-fcd3-447d-bd79-f5e147c0ed84","Handler":null,"Name":"","Endpoint":""}
Oct 31 00:18:59.989512 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:18:59.989490    4327 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 00:18:59.989606 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.989521    4327 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 00:18:59.989606 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:18:59.989574    4327 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 00:19:00.013825 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.013783    4327 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal"]
Oct 31 00:19:00.014284 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.014242    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal" event={"ID":"c35a4f872afa19338c317775c3e775c3","Type":"ContainerStarted","Data":"e8bedbeece192ade68729ec36f0da3240f18edf9a66e29ddf3401d8229657d79"}
Oct 31 00:19:00.014284 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.014283    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal" event={"ID":"c35a4f872afa19338c317775c3e775c3","Type":"ContainerDied","Data":"8b59e0cb2290f46f406233a8a8b14a37f3bba28af7d00c23a39b289387ccfa13"}
Oct 31 00:19:00.014473 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.014303    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal" event={"ID":"c35a4f872afa19338c317775c3e775c3","Type":"ContainerStarted","Data":"0bde3ead4ffd0b3a9857b98bf246d4d6ba0a513dadbbe5dbf7b13319160da364"}
Oct 31 00:19:00.014473 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.014326    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal" event={"ID":"dd74f5dcc55b4e9100e01ca71d574990","Type":"ContainerStarted","Data":"2857a4cdce295468836e3b02642f847bb3932773aac780051e7c79c8d1b79a7c"}
Oct 31 00:19:00.014473 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.014341    4327 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Oct 31 00:19:00.014473 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.014343    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal" event={"ID":"dd74f5dcc55b4e9100e01ca71d574990","Type":"ContainerStarted","Data":"5baf3e2893288f4a3179df9ca4738c5072442391df2aea0617d11ba58ce57975"}
Oct 31 00:19:00.014604 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.014458    4327 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal"
Oct 31 00:19:00.041014 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.040987    4327 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:19:00.041137 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:19:00.041037    4327 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-253.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal"
Oct 31 00:19:00.041388 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.041374    4327 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:19:00.041426 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:19:00.041415    4327 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Oct 31 00:19:00.088208 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.088182    4327 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:19:00.089596 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.089575    4327 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientMemory"
Oct 31 00:19:00.089728 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.089612    4327 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasNoDiskPressure"
Oct 31 00:19:00.089728 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.089626    4327 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientPID"
Oct 31 00:19:00.089728 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.089669    4327 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-253.ec2.internal"
Oct 31 00:19:00.130614 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.130583    4327 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-253.ec2.internal"
Oct 31 00:19:00.130790 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.130663    4327 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-253.ec2.internal"
Oct 31 00:19:00.137772 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.137743    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dd74f5dcc55b4e9100e01ca71d574990-config\") pod \"kube-apiserver-proxy-ip-10-0-128-253.ec2.internal\" (UID: \"dd74f5dcc55b4e9100e01ca71d574990\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal"
Oct 31 00:19:00.137890 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.137776    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c35a4f872afa19338c317775c3e775c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal\" (UID: \"c35a4f872afa19338c317775c3e775c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Oct 31 00:19:00.137890 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.137801    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c35a4f872afa19338c317775c3e775c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal\" (UID: \"c35a4f872afa19338c317775c3e775c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Oct 31 00:19:00.238590 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.238551    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dd74f5dcc55b4e9100e01ca71d574990-config\") pod \"kube-apiserver-proxy-ip-10-0-128-253.ec2.internal\" (UID: \"dd74f5dcc55b4e9100e01ca71d574990\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal"
Oct 31 00:19:00.238590 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.238590    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c35a4f872afa19338c317775c3e775c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal\" (UID: \"c35a4f872afa19338c317775c3e775c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Oct 31 00:19:00.238827 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.238605    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c35a4f872afa19338c317775c3e775c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal\" (UID: \"c35a4f872afa19338c317775c3e775c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Oct 31 00:19:00.238827 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.238655    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c35a4f872afa19338c317775c3e775c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal\" (UID: \"c35a4f872afa19338c317775c3e775c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Oct 31 00:19:00.238827 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.238670    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dd74f5dcc55b4e9100e01ca71d574990-config\") pod \"kube-apiserver-proxy-ip-10-0-128-253.ec2.internal\" (UID: \"dd74f5dcc55b4e9100e01ca71d574990\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal"
Oct 31 00:19:00.238827 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.238706    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c35a4f872afa19338c317775c3e775c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal\" (UID: \"c35a4f872afa19338c317775c3e775c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Oct 31 00:19:00.867444 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.867409    4327 apiserver.go:52] "Watching apiserver"
Oct 31 00:19:00.914300 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.914265    4327 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 00:19:00.914708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.914668    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-5tw4x","kube-system/konnectivity-agent-g24ff","openshift-dns/dns-default-v8ft8","openshift-ingress-canary/ingress-canary-h26l7","openshift-image-registry/node-ca-htbtd","openshift-multus/multus-additional-cni-plugins-pwt7t","openshift-network-diagnostics/network-check-target-vdxch","openshift-insights/insights-runtime-extractor-vv2qm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g","openshift-cluster-node-tuning-operator/tuned-qk5kf","openshift-network-operator/iptables-alerter-9j7z8","kube-system/global-pull-secret-syncer-pncsr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal","openshift-multus/multus-6ljg5","openshift-ovn-kubernetes/ovnkube-node-nv7t7","openshift-dns/node-resolver-9d69r","openshift-service-ca/service-ca-5fffc97bdf-lpntz"]
Oct 31 00:19:00.918138 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.918114    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:19:00.919163 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.919144    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:19:00.934489 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.934447    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rqjpt\""
Oct 31 00:19:00.934608 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.934494    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wgmpl\""
Oct 31 00:19:00.934608 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.934544    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-grldd\""
Oct 31 00:19:00.934608 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.934550    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4w2kf\""
Oct 31 00:19:00.934608 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.934599    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 00:19:00.934844 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.934632    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 00:19:00.934844 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.934711    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9prwf\""
Oct 31 00:19:00.934844 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.934717    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 00:19:00.934844 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.934795    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 00:19:00.934844 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.934796    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:19:00.934844 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.934797    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 00:19:00.934844 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.934814    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 00:19:00.934844 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.934825    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 00:19:00.934844 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.934834    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 00:19:00.935215 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.934977    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 00:19:00.935215 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.934986    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 00:19:00.935215 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.935025    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jw4zk\""
Oct 31 00:19:00.935215 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.935088    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 00:19:00.935215 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.935091    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 00:19:00.935215 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.935214    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 00:19:00.935436 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.935228    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 00:19:00.935436 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.935245    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 00:19:00.935436 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.935261    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 00:19:00.935436 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.935295    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 00:19:00.935436 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.935374    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 00:19:00.945089 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.945069    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dqh4t\""
Oct 31 00:19:00.945089 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.945083    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-6b92m\""
Oct 31 00:19:00.945778 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.945761    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 00:19:00.945831 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.945787    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 00:19:00.945908 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.945895    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 00:19:00.945950 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.945939    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 00:19:00.948134 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.948116    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:19:00.948272 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.948257    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 00:19:00.948335 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.948286    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 00:19:00.948673 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.948655    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 00:19:00.948673 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.948669    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 00:19:00.948810 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.948660    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 00:19:00.948810 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.948723    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 00:19:00.948810 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.948745    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 00:19:00.948948 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.948857    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 00:19:00.948948 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.948882    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 00:19:00.949038 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.948957    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9wq7f\""
Oct 31 00:19:00.949038 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.948978    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 00:19:00.949038 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.948987    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 00:19:00.949038 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.948981    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 00:19:00.949038 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.949013    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 00:19:00.949038 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.949018    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 00:19:00.949038 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.949013    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dsrzd\""
Oct 31 00:19:00.949038 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.949034    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 00:19:00.949327 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.949019    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-wc6r2\""
Oct 31 00:19:00.949327 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.949035    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4wlc7\""
Oct 31 00:19:00.949327 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.949073    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pp79q\""
Oct 31 00:19:00.949327 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.949074    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-l44c8\""
Oct 31 00:19:00.949327 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.949096    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 00:19:00.949327 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.949097    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5c7xv\""
Oct 31 00:19:00.949327 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.949113    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rltkk\""
Oct 31 00:19:00.949327 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.949097    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 00:19:00.949327 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.949126    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 00:19:00.949327 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.949129    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 00:19:00.949327 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.949185    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 00:19:00.952727 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.952710    4327 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal"
Oct 31 00:19:00.952831 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.952818    4327 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Oct 31 00:19:00.980925 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.980902    4327 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 00:19:00.985830 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.985816    4327 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:19:00.985891 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:19:00.985865    4327 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-253.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal"
Oct 31 00:19:00.986635 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:00.986620    4327 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:19:00.986720 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:19:00.986666    4327 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Oct 31 00:19:01.042955 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.042927    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8072caf6-1730-4100-a87a-630ace22a370-cert\") pod \"ingress-canary-h26l7\" (UID: \"8072caf6-1730-4100-a87a-630ace22a370\") " pod="openshift-ingress-canary/ingress-canary-h26l7"
Oct 31 00:19:01.042955 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.042957    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95f3be30-2d65-4373-9623-155c841dca11-host\") pod \"node-ca-htbtd\" (UID: \"95f3be30-2d65-4373-9623-155c841dca11\") " pod="openshift-image-registry/node-ca-htbtd"
Oct 31 00:19:01.043121 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.042993    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0c02c18-5891-493e-ab22-5191677b8364-ovnkube-config\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.043121 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.043020    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8190e88-1887-415f-95cb-d421f3fbc08b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:19:01.043229 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.043187    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-run-netns\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.043270 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.043225    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-socket-dir\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:19:01.043315 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.043297    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f18d4e41-daf0-4be5-bcdb-c00d189419ee-signing-key\") pod \"service-ca-5fffc97bdf-lpntz\" (UID: \"f18d4e41-daf0-4be5-bcdb-c00d189419ee\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpntz"
Oct 31 00:19:01.043467 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.043330    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ec6a6238-81be-4a82-b346-b4c9fe3aefa4-hosts-file\") pod \"node-resolver-9d69r\" (UID: \"ec6a6238-81be-4a82-b346-b4c9fe3aefa4\") " pod="openshift-dns/node-resolver-9d69r"
Oct 31 00:19:01.043467 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.043352    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-sysctl-d\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.043467 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.043378    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-lib-modules\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.043467 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.043402    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fdd0807-bb6c-4867-aad0-40af2ddc7d08-host-slash\") pod \"iptables-alerter-9j7z8\" (UID: \"4fdd0807-bb6c-4867-aad0-40af2ddc7d08\") " pod="openshift-network-operator/iptables-alerter-9j7z8"
Oct 31 00:19:01.043467 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.043399    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0c02c18-5891-493e-ab22-5191677b8364-ovnkube-config\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.043467 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.043418    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8072caf6-1730-4100-a87a-630ace22a370-cert\") pod \"ingress-canary-h26l7\" (UID: \"8072caf6-1730-4100-a87a-630ace22a370\") " pod="openshift-ingress-canary/ingress-canary-h26l7"
Oct 31 00:19:01.043467 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.043457    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-run-k8s-cni-cncf-io\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.043832 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.043399    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8190e88-1887-415f-95cb-d421f3fbc08b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:19:01.043832 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.043528    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/79ab34ed-5163-4cff-8cf4-170c3445e655-metrics-tls\") pod \"dns-default-v8ft8\" (UID: \"79ab34ed-5163-4cff-8cf4-170c3445e655\") " pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:19:01.043832 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.043577    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-kubernetes\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.043832 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.043708    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f18d4e41-daf0-4be5-bcdb-c00d189419ee-signing-key\") pod \"service-ca-5fffc97bdf-lpntz\" (UID: \"f18d4e41-daf0-4be5-bcdb-c00d189419ee\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpntz"
Oct 31 00:19:01.043942 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.043840    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/79ab34ed-5163-4cff-8cf4-170c3445e655-metrics-tls\") pod \"dns-default-v8ft8\" (UID: \"79ab34ed-5163-4cff-8cf4-170c3445e655\") " pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:19:01.044064 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.044048    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84a754a9-1c06-4302-bb01-4f53a1e032d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:19:01.044094 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.044077    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-registration-dir\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:19:01.044122 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.044099    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvxxw\" (UniqueName: \"kubernetes.io/projected/79ab34ed-5163-4cff-8cf4-170c3445e655-kube-api-access-jvxxw\") pod \"dns-default-v8ft8\" (UID: \"79ab34ed-5163-4cff-8cf4-170c3445e655\") " pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:19:01.044162 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.044118    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-run-systemd\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.044162 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.044136    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/002d2da2-bb18-4667-98c8-1770b70d9985-original-pull-secret\") pod \"global-pull-secret-syncer-pncsr\" (UID: \"002d2da2-bb18-4667-98c8-1770b70d9985\") " pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:19:01.044162 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.044151    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-var-lib-cni-bin\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.044294 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.044166    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7pbg\" (UniqueName: \"kubernetes.io/projected/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-kube-api-access-m7pbg\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.044294 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.044208    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17401e80-fe41-417f-9840-acd7a68d17e3-konnectivity-ca\") pod \"konnectivity-agent-g24ff\" (UID: \"17401e80-fe41-417f-9840-acd7a68d17e3\") " pod="kube-system/konnectivity-agent-g24ff"
Oct 31 00:19:01.044294 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.044265    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz8l7\" (UniqueName: \"kubernetes.io/projected/ec6a6238-81be-4a82-b346-b4c9fe3aefa4-kube-api-access-rz8l7\") pod \"node-resolver-9d69r\" (UID: \"ec6a6238-81be-4a82-b346-b4c9fe3aefa4\") " pod="openshift-dns/node-resolver-9d69r"
Oct 31 00:19:01.044470 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.044434    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-log-socket\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.044470 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.044431    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/002d2da2-bb18-4667-98c8-1770b70d9985-original-pull-secret\") pod \"global-pull-secret-syncer-pncsr\" (UID: \"002d2da2-bb18-4667-98c8-1770b70d9985\") " pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:19:01.044605 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.044490    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17401e80-fe41-417f-9840-acd7a68d17e3-konnectivity-ca\") pod \"konnectivity-agent-g24ff\" (UID: \"17401e80-fe41-417f-9840-acd7a68d17e3\") " pod="kube-system/konnectivity-agent-g24ff"
Oct 31 00:19:01.044605 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.044506    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-run-ovn-kubernetes\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.044605 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.044529    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-etc-kubernetes\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.044605 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.044550    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84a754a9-1c06-4302-bb01-4f53a1e032d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:19:01.044605 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.044581    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-host\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.044780 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.044640    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-cni-netd\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.044780 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.044665    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0c02c18-5891-493e-ab22-5191677b8364-ovn-node-metrics-cert\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.044780 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.044710    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0c02c18-5891-493e-ab22-5191677b8364-ovnkube-script-lib\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.044780 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.044734    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88p6w\" (UniqueName: \"kubernetes.io/projected/b0c02c18-5891-493e-ab22-5191677b8364-kube-api-access-88p6w\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.044780 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.044754    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/79ab34ed-5163-4cff-8cf4-170c3445e655-config-volume\") pod \"dns-default-v8ft8\" (UID: \"79ab34ed-5163-4cff-8cf4-170c3445e655\") " pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:19:01.044937 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.044805    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84a754a9-1c06-4302-bb01-4f53a1e032d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:19:01.044937 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.044904    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79ab34ed-5163-4cff-8cf4-170c3445e655-tmp-dir\") pod \"dns-default-v8ft8\" (UID: \"79ab34ed-5163-4cff-8cf4-170c3445e655\") " pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:19:01.045003 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.044954    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-755hv\" (UniqueName: \"kubernetes.io/projected/1f456866-ce05-42c8-adb6-588cf5fcd8b7-kube-api-access-755hv\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.045003 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.044991    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0c02c18-5891-493e-ab22-5191677b8364-ovnkube-script-lib\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.045064 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045016    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwvkk\" (UniqueName: \"kubernetes.io/projected/84a754a9-1c06-4302-bb01-4f53a1e032d0-kube-api-access-fwvkk\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:19:01.045064 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045049    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79ab34ed-5163-4cff-8cf4-170c3445e655-tmp-dir\") pod \"dns-default-v8ft8\" (UID: \"79ab34ed-5163-4cff-8cf4-170c3445e655\") " pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:19:01.045178 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045082    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:19:01.045178 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045066    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/79ab34ed-5163-4cff-8cf4-170c3445e655-config-volume\") pod \"dns-default-v8ft8\" (UID: \"79ab34ed-5163-4cff-8cf4-170c3445e655\") " pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:19:01.045178 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045119    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-metrics-certs\") pod \"network-metrics-daemon-5tw4x\" (UID: \"3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a\") " pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:19:01.045178 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045144    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-modprobe-d\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.045178 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045154    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0c02c18-5891-493e-ab22-5191677b8364-ovn-node-metrics-cert\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.045318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045193    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-sysconfig\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.045318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045218    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-run-netns\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.045318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045239    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8190e88-1887-415f-95cb-d421f3fbc08b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:19:01.045318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045280    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bbs6\" (UniqueName: \"kubernetes.io/projected/4fdd0807-bb6c-4867-aad0-40af2ddc7d08-kube-api-access-6bbs6\") pod \"iptables-alerter-9j7z8\" (UID: \"4fdd0807-bb6c-4867-aad0-40af2ddc7d08\") " pod="openshift-network-operator/iptables-alerter-9j7z8"
Oct 31 00:19:01.045318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045303    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-var-lib-kubelet\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.045471 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045324    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bwkn\" (UniqueName: \"kubernetes.io/projected/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-kube-api-access-7bwkn\") pod \"network-metrics-daemon-5tw4x\" (UID: \"3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a\") " pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:19:01.045471 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045353    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-slash\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.045471 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045389    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0c02c18-5891-493e-ab22-5191677b8364-env-overrides\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.045471 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045409    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-system-cni-dir\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.045471 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045430    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-device-dir\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:19:01.045471 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045456    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec6a6238-81be-4a82-b346-b4c9fe3aefa4-tmp-dir\") pod \"node-resolver-9d69r\" (UID: \"ec6a6238-81be-4a82-b346-b4c9fe3aefa4\") " pod="openshift-dns/node-resolver-9d69r"
Oct 31 00:19:01.045655 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045514    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8190e88-1887-415f-95cb-d421f3fbc08b-system-cni-dir\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:19:01.045655 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045525    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec6a6238-81be-4a82-b346-b4c9fe3aefa4-tmp-dir\") pod \"node-resolver-9d69r\" (UID: \"ec6a6238-81be-4a82-b346-b4c9fe3aefa4\") " pod="openshift-dns/node-resolver-9d69r"
Oct 31 00:19:01.045655 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045543    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-cnibin\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.045655 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045571    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95f3be30-2d65-4373-9623-155c841dca11-serviceca\") pod \"node-ca-htbtd\" (UID: \"95f3be30-2d65-4373-9623-155c841dca11\") " pod="openshift-image-registry/node-ca-htbtd"
Oct 31 00:19:01.045655 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045586    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-sysctl-conf\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.045655 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045603    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-run\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.045655 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045618    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-sys\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.045876 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045657    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-var-lib-kubelet\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.045876 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045658    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0c02c18-5891-493e-ab22-5191677b8364-env-overrides\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.045876 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045701    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f456866-ce05-42c8-adb6-588cf5fcd8b7-tmp\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.045876 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045726    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-multus-daemon-config\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.045876 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045742    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mndzv\" (UniqueName: \"kubernetes.io/projected/95f3be30-2d65-4373-9623-155c841dca11-kube-api-access-mndzv\") pod \"node-ca-htbtd\" (UID: \"95f3be30-2d65-4373-9623-155c841dca11\") " pod="openshift-image-registry/node-ca-htbtd"
Oct 31 00:19:01.045876 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045757    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f456866-ce05-42c8-adb6-588cf5fcd8b7-tmp\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.045876 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045758    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-kubelet\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.045876 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045793    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-run-openvswitch\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.045876 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045824    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-node-log\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.045876 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045827    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95f3be30-2d65-4373-9623-155c841dca11-serviceca\") pod \"node-ca-htbtd\" (UID: \"95f3be30-2d65-4373-9623-155c841dca11\") " pod="openshift-image-registry/node-ca-htbtd"
Oct 31 00:19:01.045876 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045853    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.046210 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045891    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8190e88-1887-415f-95cb-d421f3fbc08b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:19:01.046210 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045920    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk4vp\" (UniqueName: \"kubernetes.io/projected/a8190e88-1887-415f-95cb-d421f3fbc08b-kube-api-access-fk4vp\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:19:01.046210 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045960    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-multus-socket-dir-parent\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.046210 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.045980    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-multus-conf-dir\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.046210 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046014    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-run-multus-certs\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.046210 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046048    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f18d4e41-daf0-4be5-bcdb-c00d189419ee-signing-cabundle\") pod \"service-ca-5fffc97bdf-lpntz\" (UID: \"f18d4e41-daf0-4be5-bcdb-c00d189419ee\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpntz"
Oct 31 00:19:01.046210 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046092    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-multus-daemon-config\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.046210 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046135    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8190e88-1887-415f-95cb-d421f3fbc08b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:19:01.046210 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046150    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-systemd-units\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.046210 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046180    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-var-lib-cni-multus\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.046210 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046190    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f18d4e41-daf0-4be5-bcdb-c00d189419ee-signing-cabundle\") pod \"service-ca-5fffc97bdf-lpntz\" (UID: \"f18d4e41-daf0-4be5-bcdb-c00d189419ee\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpntz"
Oct 31 00:19:01.046210 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046196    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-hostroot\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.046599 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046225    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-sys-fs\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:19:01.046599 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046242    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-etc-openvswitch\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.046599 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046280    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4fdd0807-bb6c-4867-aad0-40af2ddc7d08-iptables-alerter-script\") pod \"iptables-alerter-9j7z8\" (UID: \"4fdd0807-bb6c-4867-aad0-40af2ddc7d08\") " pod="openshift-network-operator/iptables-alerter-9j7z8"
Oct 31 00:19:01.046599 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046313    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngkbc\" (UniqueName: \"kubernetes.io/projected/5b9f4a03-68eb-418e-9447-ad62473a1e9a-kube-api-access-ngkbc\") pod \"network-check-target-vdxch\" (UID: \"5b9f4a03-68eb-418e-9447-ad62473a1e9a\") " pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:19:01.046599 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046329    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-multus-cni-dir\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.046599 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046364    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17401e80-fe41-417f-9840-acd7a68d17e3-agent-certs\") pod \"konnectivity-agent-g24ff\" (UID: \"17401e80-fe41-417f-9840-acd7a68d17e3\") " pod="kube-system/konnectivity-agent-g24ff"
Oct 31 00:19:01.046599 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046404    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvblv\" (UniqueName: \"kubernetes.io/projected/f18d4e41-daf0-4be5-bcdb-c00d189419ee-kube-api-access-bvblv\") pod \"service-ca-5fffc97bdf-lpntz\" (UID: \"f18d4e41-daf0-4be5-bcdb-c00d189419ee\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpntz"
Oct 31 00:19:01.046599 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046541    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-systemd\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.046599 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046548    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4fdd0807-bb6c-4867-aad0-40af2ddc7d08-iptables-alerter-script\") pod \"iptables-alerter-9j7z8\" (UID: \"4fdd0807-bb6c-4867-aad0-40af2ddc7d08\") " pod="openshift-network-operator/iptables-alerter-9j7z8"
Oct 31 00:19:01.046599 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046576    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-tuned\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.047040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046605    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-run-ovn\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.047040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046665    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-tuned\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.047040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046666    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84a754a9-1c06-4302-bb01-4f53a1e032d0-data-volume\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:19:01.047040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046707    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17401e80-fe41-417f-9840-acd7a68d17e3-agent-certs\") pod \"konnectivity-agent-g24ff\" (UID: \"17401e80-fe41-417f-9840-acd7a68d17e3\") " pod="kube-system/konnectivity-agent-g24ff"
Oct 31 00:19:01.047040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046742    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q65jt\" (UniqueName: \"kubernetes.io/projected/8072caf6-1730-4100-a87a-630ace22a370-kube-api-access-q65jt\") pod \"ingress-canary-h26l7\" (UID: \"8072caf6-1730-4100-a87a-630ace22a370\") " pod="openshift-ingress-canary/ingress-canary-h26l7"
Oct 31 00:19:01.047040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046738    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84a754a9-1c06-4302-bb01-4f53a1e032d0-data-volume\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:19:01.047040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046768    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-etc-selinux\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:19:01.047040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046794    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkmrt\" (UniqueName: \"kubernetes.io/projected/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-kube-api-access-mkmrt\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:19:01.047040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046833    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-var-lib-openvswitch\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.047040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046867    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-cni-bin\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.047040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046892    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8190e88-1887-415f-95cb-d421f3fbc08b-cnibin\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:19:01.047040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046922    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8190e88-1887-415f-95cb-d421f3fbc08b-cni-binary-copy\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:19:01.047040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.046952    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/002d2da2-bb18-4667-98c8-1770b70d9985-dbus\") pod \"global-pull-secret-syncer-pncsr\" (UID: \"002d2da2-bb18-4667-98c8-1770b70d9985\") " pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:19:01.047040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.047008    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84a754a9-1c06-4302-bb01-4f53a1e032d0-crio-socket\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:19:01.047416 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.047065    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-cni-binary-copy\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.047416 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.047095    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8190e88-1887-415f-95cb-d421f3fbc08b-cni-binary-copy\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:19:01.047416 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.047101    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/002d2da2-bb18-4667-98c8-1770b70d9985-kubelet-config\") pod \"global-pull-secret-syncer-pncsr\" (UID: \"002d2da2-bb18-4667-98c8-1770b70d9985\") " pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:19:01.047416 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.047231    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-cni-binary-copy\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.069937 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.069915    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88p6w\" (UniqueName: \"kubernetes.io/projected/b0c02c18-5891-493e-ab22-5191677b8364-kube-api-access-88p6w\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.070073 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.070057    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwvkk\" (UniqueName: \"kubernetes.io/projected/84a754a9-1c06-4302-bb01-4f53a1e032d0-kube-api-access-fwvkk\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:19:01.070110 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.070063    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7pbg\" (UniqueName: \"kubernetes.io/projected/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-kube-api-access-m7pbg\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.070545 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.070510    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvxxw\" (UniqueName: \"kubernetes.io/projected/79ab34ed-5163-4cff-8cf4-170c3445e655-kube-api-access-jvxxw\") pod \"dns-default-v8ft8\" (UID: \"79ab34ed-5163-4cff-8cf4-170c3445e655\") " pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:19:01.070774 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.070758    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz8l7\" (UniqueName: \"kubernetes.io/projected/ec6a6238-81be-4a82-b346-b4c9fe3aefa4-kube-api-access-rz8l7\") pod \"node-resolver-9d69r\" (UID: \"ec6a6238-81be-4a82-b346-b4c9fe3aefa4\") " pod="openshift-dns/node-resolver-9d69r"
Oct 31 00:19:01.074767 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.074751    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-755hv\" (UniqueName: \"kubernetes.io/projected/1f456866-ce05-42c8-adb6-588cf5fcd8b7-kube-api-access-755hv\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.076596 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.076579    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkmrt\" (UniqueName: \"kubernetes.io/projected/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-kube-api-access-mkmrt\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:19:01.077656 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.077635    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngkbc\" (UniqueName: \"kubernetes.io/projected/5b9f4a03-68eb-418e-9447-ad62473a1e9a-kube-api-access-ngkbc\") pod \"network-check-target-vdxch\" (UID: \"5b9f4a03-68eb-418e-9447-ad62473a1e9a\") " pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:19:01.077745 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.077702    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bbs6\" (UniqueName: \"kubernetes.io/projected/4fdd0807-bb6c-4867-aad0-40af2ddc7d08-kube-api-access-6bbs6\") pod \"iptables-alerter-9j7z8\" (UID: \"4fdd0807-bb6c-4867-aad0-40af2ddc7d08\") " pod="openshift-network-operator/iptables-alerter-9j7z8"
Oct 31 00:19:01.081397 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.081381    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk4vp\" (UniqueName: \"kubernetes.io/projected/a8190e88-1887-415f-95cb-d421f3fbc08b-kube-api-access-fk4vp\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:19:01.081966 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.081952    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bwkn\" (UniqueName: \"kubernetes.io/projected/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-kube-api-access-7bwkn\") pod \"network-metrics-daemon-5tw4x\" (UID: \"3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a\") " pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:19:01.082167 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.082145    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mndzv\" (UniqueName: \"kubernetes.io/projected/95f3be30-2d65-4373-9623-155c841dca11-kube-api-access-mndzv\") pod \"node-ca-htbtd\" (UID: \"95f3be30-2d65-4373-9623-155c841dca11\") " pod="openshift-image-registry/node-ca-htbtd"
Oct 31 00:19:01.082413 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.082396    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q65jt\" (UniqueName: \"kubernetes.io/projected/8072caf6-1730-4100-a87a-630ace22a370-kube-api-access-q65jt\") pod \"ingress-canary-h26l7\" (UID: \"8072caf6-1730-4100-a87a-630ace22a370\") " pod="openshift-ingress-canary/ingress-canary-h26l7"
Oct 31 00:19:01.082730 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.082714    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvblv\" (UniqueName: \"kubernetes.io/projected/f18d4e41-daf0-4be5-bcdb-c00d189419ee-kube-api-access-bvblv\") pod \"service-ca-5fffc97bdf-lpntz\" (UID: \"f18d4e41-daf0-4be5-bcdb-c00d189419ee\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpntz"
Oct 31 00:19:01.148042 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148012    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-kubernetes\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.148163 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148046    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84a754a9-1c06-4302-bb01-4f53a1e032d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:19:01.148163 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148067    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-registration-dir\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:19:01.148163 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148093    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-run-systemd\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.148163 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148116    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-var-lib-cni-bin\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.148163 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148141    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-log-socket\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.148336 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148162    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-run-ovn-kubernetes\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.148336 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148164    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-kubernetes\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.148336 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148184    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-etc-kubernetes\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.148336 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148226    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-host\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.148336 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148253    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-cni-netd\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.148336 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148267    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-registration-dir\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:19:01.148336 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148284    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:19:01.148336 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148309    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-metrics-certs\") pod \"network-metrics-daemon-5tw4x\" (UID: \"3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a\") " pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:19:01.148336 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148331    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-modprobe-d\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.148708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148338    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-var-lib-cni-bin\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.148708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148353    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-sysconfig\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.148708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148377    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-run-netns\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.148708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148404    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8190e88-1887-415f-95cb-d421f3fbc08b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:19:01.148708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148409    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-run-ovn-kubernetes\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.148708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148309    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-run-systemd\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.148708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148429    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-var-lib-kubelet\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.148708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148453    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-slash\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.148708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148462    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-host\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.148708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148487    4327 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 00:19:01.148708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148504    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-slash\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.148708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148519    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-sysconfig\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.148708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148512    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-cni-netd\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.148708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148543    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:19:01.148708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148561    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-system-cni-dir\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.148708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148603    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-device-dir\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:19:01.148708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148612    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8190e88-1887-415f-95cb-d421f3fbc08b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:19:01.148708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148641    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-modprobe-d\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.148708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148650    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8190e88-1887-415f-95cb-d421f3fbc08b-system-cni-dir\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:19:01.148708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148689    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-var-lib-kubelet\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.148708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148698    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-cnibin\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.148708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148378    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-log-socket\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148228    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-etc-kubernetes\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148744    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8190e88-1887-415f-95cb-d421f3fbc08b-system-cni-dir\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148567    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-run-netns\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148787    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-system-cni-dir\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148787    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-cnibin\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148808    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-sysctl-conf\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148840    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-run\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148858    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-device-dir\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148862    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-sys\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148898    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-var-lib-kubelet\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148908    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-sys\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148941    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-kubelet\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148956    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-run\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148964    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-run-openvswitch\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.148988    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-node-log\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149029    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149035    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-kubelet\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149025    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-sysctl-conf\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149062    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-multus-socket-dir-parent\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149081    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-run-openvswitch\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149100    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149102    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-multus-conf-dir\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149115    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-node-log\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149130    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-run-multus-certs\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149151    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-systemd-units\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149145    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-multus-conf-dir\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149169    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-var-lib-cni-multus\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149003    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-var-lib-kubelet\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149172    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-multus-socket-dir-parent\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149184    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-hostroot\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149203    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-run-multus-certs\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149211    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-var-lib-cni-multus\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149221    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-systemd-units\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149239    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-sys-fs\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149269    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-etc-openvswitch\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149279    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-sys-fs\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149295    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-multus-cni-dir\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149314    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-hostroot\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149317    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-etc-openvswitch\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149335    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-systemd\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149361    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-run-ovn\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149377    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-multus-cni-dir\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149399    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-etc-selinux\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149407    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-systemd\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149422    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-run-ovn\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149424    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-var-lib-openvswitch\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149449    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-cni-bin\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149443    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-etc-selinux\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149488    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8190e88-1887-415f-95cb-d421f3fbc08b-cnibin\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149499    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-var-lib-openvswitch\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149514    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/002d2da2-bb18-4667-98c8-1770b70d9985-dbus\") pod \"global-pull-secret-syncer-pncsr\" (UID: \"002d2da2-bb18-4667-98c8-1770b70d9985\") " pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149526    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0c02c18-5891-493e-ab22-5191677b8364-host-cni-bin\") pod \"ovnkube-node-nv7t7\" (UID: \"b0c02c18-5891-493e-ab22-5191677b8364\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149554    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84a754a9-1c06-4302-bb01-4f53a1e032d0-crio-socket\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149581    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/002d2da2-bb18-4667-98c8-1770b70d9985-kubelet-config\") pod \"global-pull-secret-syncer-pncsr\" (UID: \"002d2da2-bb18-4667-98c8-1770b70d9985\") " pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149604    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95f3be30-2d65-4373-9623-155c841dca11-host\") pod \"node-ca-htbtd\" (UID: \"95f3be30-2d65-4373-9623-155c841dca11\") " pod="openshift-image-registry/node-ca-htbtd"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149631    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/002d2da2-bb18-4667-98c8-1770b70d9985-dbus\") pod \"global-pull-secret-syncer-pncsr\" (UID: \"002d2da2-bb18-4667-98c8-1770b70d9985\") " pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149558    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8190e88-1887-415f-95cb-d421f3fbc08b-cnibin\") pod \"multus-additional-cni-plugins-pwt7t\" (UID: \"a8190e88-1887-415f-95cb-d421f3fbc08b\") " pod="openshift-multus/multus-additional-cni-plugins-pwt7t"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149640    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-run-netns\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149663    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-socket-dir\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149699    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-run-netns\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149705    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ec6a6238-81be-4a82-b346-b4c9fe3aefa4-hosts-file\") pod \"node-resolver-9d69r\" (UID: \"ec6a6238-81be-4a82-b346-b4c9fe3aefa4\") " pod="openshift-dns/node-resolver-9d69r"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149725    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-sysctl-d\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149747    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-lib-modules\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149756    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/002d2da2-bb18-4667-98c8-1770b70d9985-kubelet-config\") pod \"global-pull-secret-syncer-pncsr\" (UID: \"002d2da2-bb18-4667-98c8-1770b70d9985\") " pod="kube-system/global-pull-secret-syncer-pncsr"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149761    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fdd0807-bb6c-4867-aad0-40af2ddc7d08-host-slash\") pod \"iptables-alerter-9j7z8\" (UID: \"4fdd0807-bb6c-4867-aad0-40af2ddc7d08\") " pod="openshift-network-operator/iptables-alerter-9j7z8"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149787    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-run-k8s-cni-cncf-io\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.149742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149787    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fdd0807-bb6c-4867-aad0-40af2ddc7d08-host-slash\") pod \"iptables-alerter-9j7z8\" (UID: \"4fdd0807-bb6c-4867-aad0-40af2ddc7d08\") " pod="openshift-network-operator/iptables-alerter-9j7z8"
Oct 31 00:19:01.151994 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149842    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95f3be30-2d65-4373-9623-155c841dca11-host\") pod \"node-ca-htbtd\" (UID: \"95f3be30-2d65-4373-9623-155c841dca11\") " pod="openshift-image-registry/node-ca-htbtd"
Oct 31 00:19:01.151994 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149866    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bbc0fbb5-0386-40b9-aeef-3b7aa867f625-host-run-k8s-cni-cncf-io\") pod \"multus-6ljg5\" (UID: \"bbc0fbb5-0386-40b9-aeef-3b7aa867f625\") " pod="openshift-multus/multus-6ljg5"
Oct 31 00:19:01.151994 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149907    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84a754a9-1c06-4302-bb01-4f53a1e032d0-crio-socket\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:19:01.151994 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149915    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ec6a6238-81be-4a82-b346-b4c9fe3aefa4-hosts-file\") pod \"node-resolver-9d69r\" (UID: \"ec6a6238-81be-4a82-b346-b4c9fe3aefa4\") " pod="openshift-dns/node-resolver-9d69r"
Oct 31 00:19:01.151994 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149954    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/165b1a7a-d8ce-49d4-9a08-95a3cbea4f49-socket-dir\") pod \"aws-ebs-csi-driver-node-x9l2g\" (UID: \"165b1a7a-d8ce-49d4-9a08-95a3cbea4f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l2g"
Oct 31 00:19:01.151994 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.149996    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-lib-modules\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.151994 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.150017    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f456866-ce05-42c8-adb6-588cf5fcd8b7-etc-sysctl-d\") pod \"tuned-qk5kf\" (UID: \"1f456866-ce05-42c8-adb6-588cf5fcd8b7\") " pod="openshift-cluster-node-tuning-operator/tuned-qk5kf"
Oct 31 00:19:01.152335 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.152133    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84a754a9-1c06-4302-bb01-4f53a1e032d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vv2qm\" (UID: \"84a754a9-1c06-4302-bb01-4f53a1e032d0\") " pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:19:01.153456 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.153435    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a-metrics-certs\") pod \"network-metrics-daemon-5tw4x\" (UID: \"3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a\") " pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:19:01.218530 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.218494    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5tw4x"
Oct 31 00:19:01.219729 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.219708    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vv2qm"
Oct 31 00:19:01.362438 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.362402    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:19:01.573244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.573175    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5tw4x"]
Oct 31 00:19:01.575585 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:19:01.575388    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f78a950_9d24_4f68_9ed3_4fec0cf5dd6a.slice/crio-e6a4d0d698dc85083ab8082ad58840e12d096618d303c0b26806c3df43549eb7 WatchSource:0}: Error finding container e6a4d0d698dc85083ab8082ad58840e12d096618d303c0b26806c3df43549eb7: Status 404 returned error can't find the container with id e6a4d0d698dc85083ab8082ad58840e12d096618d303c0b26806c3df43549eb7
Oct 31 00:19:01.576223 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.576203    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vv2qm"]
Oct 31 00:19:01.577802 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.577784    4327 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:19:01.577890 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:19:01.577855    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a754a9_1c06_4302_bb01_4f53a1e032d0.slice/crio-356ce3dd4205cacba986fc025af23769d2f7a76773bf26a23cf50346c03cf107 WatchSource:0}: Error finding container 356ce3dd4205cacba986fc025af23769d2f7a76773bf26a23cf50346c03cf107: Status 404 returned error can't find the container with id 356ce3dd4205cacba986fc025af23769d2f7a76773bf26a23cf50346c03cf107
Oct 31 00:19:01.958099 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.958060    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vv2qm" event={"ID":"84a754a9-1c06-4302-bb01-4f53a1e032d0","Type":"ContainerStarted","Data":"34f3d0a77398191c0724203f525cd4fd0f665c8becdec3afad6dffd6113edb3d"}
Oct 31 00:19:01.958551 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.958119    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vv2qm" event={"ID":"84a754a9-1c06-4302-bb01-4f53a1e032d0","Type":"ContainerStarted","Data":"356ce3dd4205cacba986fc025af23769d2f7a76773bf26a23cf50346c03cf107"}
Oct 31 00:19:01.959561 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:01.959532    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5tw4x" event={"ID":"3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a","Type":"ContainerStarted","Data":"e6a4d0d698dc85083ab8082ad58840e12d096618d303c0b26806c3df43549eb7"}
Oct 31 00:19:02.966015 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:02.965787    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vv2qm" event={"ID":"84a754a9-1c06-4302-bb01-4f53a1e032d0","Type":"ContainerStarted","Data":"327bd26c246a49b8c524d289dc126c524ca11124aefa53783893b00a6aa6bf3d"}
Oct 31 00:19:02.967976 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:02.967939    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5tw4x" event={"ID":"3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a","Type":"ContainerStarted","Data":"e0b784f39eec14cc2a04cf68cb1017f1c66c3e8cd3914ca2275230d06588e1a3"}
Oct 31 00:19:02.968094 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:02.968002    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5tw4x" event={"ID":"3f78a950-9d24-4f68-9ed3-4fec0cf5dd6a","Type":"ContainerStarted","Data":"f05cd5ee7c29ce6f2e3b4eb5fbb541881c1ca7ca455f57728d7634af4f9c5c45"}
Oct 31 00:19:03.190017 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:03.189977    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:19:03.280734 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:03.280651    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:03.974261 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:03.974225    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vv2qm" event={"ID":"84a754a9-1c06-4302-bb01-4f53a1e032d0","Type":"ContainerStarted","Data":"3a69bf2e2919eefd227a2e9a8c9fdd0957cf1050ed354263c9ecae2bcaae5ffb"}
Oct 31 00:19:08.081084 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:08.081048    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:08.081488 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:08.081259    4327 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 00:19:09.192631 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:09.192604    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-v8ft8"
Oct 31 00:19:11.942043 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:11.942011    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j9k2n"]
Oct 31 00:19:11.944947 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:11.944915    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j9k2n"
Oct 31 00:19:11.956202 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:11.956151    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-4k28w"]
Oct 31 00:19:11.961830 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:11.961810    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-4k28w"
Oct 31 00:19:11.971671 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:11.971647    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vt755\""
Oct 31 00:19:11.972369 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:11.972345    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j9k2n"]
Oct 31 00:19:11.983596 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:11.983569    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 31 00:19:11.983596 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:11.983589    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 00:19:11.984085 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:11.984067    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-4k28w"]
Oct 31 00:19:11.984873 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:11.984861    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 00:19:12.081748 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.081705    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-564986b4bd-526b7"]
Oct 31 00:19:12.084913 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.084889    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:12.094976 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.094954    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-w9lxd\""
Oct 31 00:19:12.115565 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.115534    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93aea69b-c34d-4a21-acf8-a71d62fe8ef2-tmp\") pod \"downloads-6dd789c8b7-4k28w\" (UID: \"93aea69b-c34d-4a21-acf8-a71d62fe8ef2\") " pod="openshift-console/downloads-6dd789c8b7-4k28w"
Oct 31 00:19:12.115745 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.115585    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljrzz\" (UniqueName: \"kubernetes.io/projected/93aea69b-c34d-4a21-acf8-a71d62fe8ef2-kube-api-access-ljrzz\") pod \"downloads-6dd789c8b7-4k28w\" (UID: \"93aea69b-c34d-4a21-acf8-a71d62fe8ef2\") " pod="openshift-console/downloads-6dd789c8b7-4k28w"
Oct 31 00:19:12.115745 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.115606    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/dc8f439a-7709-48f9-948a-eb6b76c117e1-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-j9k2n\" (UID: \"dc8f439a-7709-48f9-948a-eb6b76c117e1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j9k2n"
Oct 31 00:19:12.141856 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.141824    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 00:19:12.142281 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.142260    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-564986b4bd-526b7"]
Oct 31 00:19:12.170808 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.170772    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 00:19:12.172117 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.172097    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 00:19:12.172211 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.172100    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-94d5g\""
Oct 31 00:19:12.172880 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.172865    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 00:19:12.216493 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.216422    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f-ca-trust-extracted\") pod \"image-registry-564986b4bd-526b7\" (UID: \"e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f\") " pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:12.216493 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.216463    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljrzz\" (UniqueName: \"kubernetes.io/projected/93aea69b-c34d-4a21-acf8-a71d62fe8ef2-kube-api-access-ljrzz\") pod \"downloads-6dd789c8b7-4k28w\" (UID: \"93aea69b-c34d-4a21-acf8-a71d62fe8ef2\") " pod="openshift-console/downloads-6dd789c8b7-4k28w"
Oct 31 00:19:12.216493 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.216484    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/dc8f439a-7709-48f9-948a-eb6b76c117e1-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-j9k2n\" (UID: \"dc8f439a-7709-48f9-948a-eb6b76c117e1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j9k2n"
Oct 31 00:19:12.216733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.216567    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f-registry-certificates\") pod \"image-registry-564986b4bd-526b7\" (UID: \"e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f\") " pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:12.216733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.216604    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f-trusted-ca\") pod \"image-registry-564986b4bd-526b7\" (UID: \"e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f\") " pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:12.216733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.216632    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93aea69b-c34d-4a21-acf8-a71d62fe8ef2-tmp\") pod \"downloads-6dd789c8b7-4k28w\" (UID: \"93aea69b-c34d-4a21-acf8-a71d62fe8ef2\") " pod="openshift-console/downloads-6dd789c8b7-4k28w"
Oct 31 00:19:12.216733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.216649    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f-registry-tls\") pod \"image-registry-564986b4bd-526b7\" (UID: \"e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f\") " pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:12.216733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.216667    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f-installation-pull-secrets\") pod \"image-registry-564986b4bd-526b7\" (UID: \"e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f\") " pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:12.216733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.216722    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t87df\" (UniqueName: \"kubernetes.io/projected/e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f-kube-api-access-t87df\") pod \"image-registry-564986b4bd-526b7\" (UID: \"e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f\") " pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:12.216982 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.216836    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f-image-registry-private-configuration\") pod \"image-registry-564986b4bd-526b7\" (UID: \"e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f\") " pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:12.216982 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.216871    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f-bound-sa-token\") pod \"image-registry-564986b4bd-526b7\" (UID: \"e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f\") " pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:12.217045 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.217020    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93aea69b-c34d-4a21-acf8-a71d62fe8ef2-tmp\") pod \"downloads-6dd789c8b7-4k28w\" (UID: \"93aea69b-c34d-4a21-acf8-a71d62fe8ef2\") " pod="openshift-console/downloads-6dd789c8b7-4k28w"
Oct 31 00:19:12.220266 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.220241    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/dc8f439a-7709-48f9-948a-eb6b76c117e1-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-j9k2n\" (UID: \"dc8f439a-7709-48f9-948a-eb6b76c117e1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j9k2n"
Oct 31 00:19:12.262938 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.262900    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j9k2n"
Oct 31 00:19:12.267507 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.267482    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljrzz\" (UniqueName: \"kubernetes.io/projected/93aea69b-c34d-4a21-acf8-a71d62fe8ef2-kube-api-access-ljrzz\") pod \"downloads-6dd789c8b7-4k28w\" (UID: \"93aea69b-c34d-4a21-acf8-a71d62fe8ef2\") " pod="openshift-console/downloads-6dd789c8b7-4k28w"
Oct 31 00:19:12.275903 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.275877    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-4k28w"
Oct 31 00:19:12.319112 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.317777    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f-ca-trust-extracted\") pod \"image-registry-564986b4bd-526b7\" (UID: \"e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f\") " pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:12.319112 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.317865    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f-registry-certificates\") pod \"image-registry-564986b4bd-526b7\" (UID: \"e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f\") " pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:12.319112 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.317893    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f-trusted-ca\") pod \"image-registry-564986b4bd-526b7\" (UID: \"e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f\") " pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:12.319112 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.317921    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f-registry-tls\") pod \"image-registry-564986b4bd-526b7\" (UID: \"e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f\") " pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:12.319112 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.317944    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f-installation-pull-secrets\") pod \"image-registry-564986b4bd-526b7\" (UID: \"e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f\") " pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:12.319112 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.317967    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t87df\" (UniqueName: \"kubernetes.io/projected/e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f-kube-api-access-t87df\") pod \"image-registry-564986b4bd-526b7\" (UID: \"e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f\") " pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:12.319112 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.318001    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f-image-registry-private-configuration\") pod \"image-registry-564986b4bd-526b7\" (UID: \"e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f\") " pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:12.319112 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.318023    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f-bound-sa-token\") pod \"image-registry-564986b4bd-526b7\" (UID: \"e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f\") " pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:12.319112 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.318226    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f-ca-trust-extracted\") pod \"image-registry-564986b4bd-526b7\" (UID: \"e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f\") " pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:12.320219 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.320125    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f-registry-certificates\") pod \"image-registry-564986b4bd-526b7\" (UID: \"e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f\") " pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:12.321154 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.321130    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f-trusted-ca\") pod \"image-registry-564986b4bd-526b7\" (UID: \"e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f\") " pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:12.322940 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.322603    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f-image-registry-private-configuration\") pod \"image-registry-564986b4bd-526b7\" (UID: \"e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f\") " pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:12.322940 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.322903    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f-installation-pull-secrets\") pod \"image-registry-564986b4bd-526b7\" (UID: \"e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f\") " pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:12.323055 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.323008    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f-registry-tls\") pod \"image-registry-564986b4bd-526b7\" (UID: \"e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f\") " pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:12.346617 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.346580    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t87df\" (UniqueName: \"kubernetes.io/projected/e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f-kube-api-access-t87df\") pod \"image-registry-564986b4bd-526b7\" (UID: \"e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f\") " pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:12.347071 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.347048    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f-bound-sa-token\") pod \"image-registry-564986b4bd-526b7\" (UID: \"e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f\") " pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:12.396186 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.396153    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:12.559509 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.559484    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j9k2n"]
Oct 31 00:19:12.561721 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:19:12.561698    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8f439a_7709_48f9_948a_eb6b76c117e1.slice/crio-eaf11e009812049256deccfb5f879348f3c4bd3066b6fbbc93d9dbafca731112 WatchSource:0}: Error finding container eaf11e009812049256deccfb5f879348f3c4bd3066b6fbbc93d9dbafca731112: Status 404 returned error can't find the container with id eaf11e009812049256deccfb5f879348f3c4bd3066b6fbbc93d9dbafca731112
Oct 31 00:19:12.597389 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.597364    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-4k28w"]
Oct 31 00:19:12.598989 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:19:12.598963    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93aea69b_c34d_4a21_acf8_a71d62fe8ef2.slice/crio-170df0879a4b5585cac2af7fc8c3e97b3b2128808c8b6940f4f40a67895331b8 WatchSource:0}: Error finding container 170df0879a4b5585cac2af7fc8c3e97b3b2128808c8b6940f4f40a67895331b8: Status 404 returned error can't find the container with id 170df0879a4b5585cac2af7fc8c3e97b3b2128808c8b6940f4f40a67895331b8
Oct 31 00:19:12.694605 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:12.694575    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-564986b4bd-526b7"]
Oct 31 00:19:12.695642 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:19:12.695617    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1b3a4a9_bed0_4574_9b0b_35c1ea45a49f.slice/crio-e81edb27b18ea74b0fd0e4da103fc40352a161976c126b77d456d38cdf0a6c70 WatchSource:0}: Error finding container e81edb27b18ea74b0fd0e4da103fc40352a161976c126b77d456d38cdf0a6c70: Status 404 returned error can't find the container with id e81edb27b18ea74b0fd0e4da103fc40352a161976c126b77d456d38cdf0a6c70
Oct 31 00:19:13.001431 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:13.001389    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-4k28w" event={"ID":"93aea69b-c34d-4a21-acf8-a71d62fe8ef2","Type":"ContainerStarted","Data":"170df0879a4b5585cac2af7fc8c3e97b3b2128808c8b6940f4f40a67895331b8"}
Oct 31 00:19:13.002866 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:13.002832    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j9k2n" event={"ID":"dc8f439a-7709-48f9-948a-eb6b76c117e1","Type":"ContainerStarted","Data":"eaf11e009812049256deccfb5f879348f3c4bd3066b6fbbc93d9dbafca731112"}
Oct 31 00:19:13.004464 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:13.004438    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-564986b4bd-526b7" event={"ID":"e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f","Type":"ContainerStarted","Data":"0b660800c9605c98347cf2c7ea177c5e35db274383c3b6d6a6944fa7461c34fc"}
Oct 31 00:19:13.004581 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:13.004482    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-564986b4bd-526b7" event={"ID":"e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f","Type":"ContainerStarted","Data":"e81edb27b18ea74b0fd0e4da103fc40352a161976c126b77d456d38cdf0a6c70"}
Oct 31 00:19:13.004655 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:13.004588    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:14.010419 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:14.010378    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j9k2n" event={"ID":"dc8f439a-7709-48f9-948a-eb6b76c117e1","Type":"ContainerStarted","Data":"40344c38d5b5f842e1d8408e54f89b146765ac6baf0fde0d40bbbbeee8548561"}
Oct 31 00:19:14.054509 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:14.054343    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-564986b4bd-526b7" podStartSLOduration=2.054321661 podStartE2EDuration="2.054321661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:13.058610806 +0000 UTC m=+13.280646351" watchObservedRunningTime="2026-10-31 00:19:14.054321661 +0000 UTC m=+14.276357201"
Oct 31 00:19:14.054733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:14.054657    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j9k2n" podStartSLOduration=2.050337807 podStartE2EDuration="3.054650838s" totalImagesPullingTime="1.004313031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:14.054111612 +0000 UTC m=+14.276147156" watchObservedRunningTime="2026-10-31 00:19:14.054650838 +0000 UTC m=+14.276686381"
Oct 31 00:19:14.109276 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:14.109243    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nv7t7"
Oct 31 00:19:15.014131 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:15.014095    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j9k2n"
Oct 31 00:19:15.019414 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:15.019387    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j9k2n"
Oct 31 00:19:15.295244 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:19:15.295147    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:19:16.031004 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:16.030962    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-srmtb"]
Oct 31 00:19:16.034384 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:16.034356    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-srmtb"
Oct 31 00:19:16.055448 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:16.055421    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 31 00:19:16.058021 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:16.057997    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 31 00:19:16.072262 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:16.072232    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 00:19:16.072262 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:16.072232    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 00:19:16.073336 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:16.073311    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-srmtb"]
Oct 31 00:19:16.083893 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:16.083865    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 00:19:16.145988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:16.145941    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-459sd\" (UniqueName: \"kubernetes.io/projected/6e3c9fbf-af3d-49e2-8a7e-0848b39fc181-kube-api-access-459sd\") pod \"prometheus-operator-9cd89b74c-srmtb\" (UID: \"6e3c9fbf-af3d-49e2-8a7e-0848b39fc181\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srmtb"
Oct 31 00:19:16.145988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:16.145992    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e3c9fbf-af3d-49e2-8a7e-0848b39fc181-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-srmtb\" (UID: \"6e3c9fbf-af3d-49e2-8a7e-0848b39fc181\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srmtb"
Oct 31 00:19:16.146199 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:16.146031    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e3c9fbf-af3d-49e2-8a7e-0848b39fc181-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-srmtb\" (UID: \"6e3c9fbf-af3d-49e2-8a7e-0848b39fc181\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srmtb"
Oct 31 00:19:16.146429 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:16.146395    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e3c9fbf-af3d-49e2-8a7e-0848b39fc181-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-srmtb\" (UID: \"6e3c9fbf-af3d-49e2-8a7e-0848b39fc181\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srmtb"
Oct 31 00:19:16.155377 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:16.155347    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mqpqj\""
Oct 31 00:19:16.246979 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:16.246939    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-459sd\" (UniqueName: \"kubernetes.io/projected/6e3c9fbf-af3d-49e2-8a7e-0848b39fc181-kube-api-access-459sd\") pod \"prometheus-operator-9cd89b74c-srmtb\" (UID: \"6e3c9fbf-af3d-49e2-8a7e-0848b39fc181\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srmtb"
Oct 31 00:19:16.247147 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:16.246994    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e3c9fbf-af3d-49e2-8a7e-0848b39fc181-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-srmtb\" (UID: \"6e3c9fbf-af3d-49e2-8a7e-0848b39fc181\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srmtb"
Oct 31 00:19:16.247147 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:16.247037    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e3c9fbf-af3d-49e2-8a7e-0848b39fc181-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-srmtb\" (UID: \"6e3c9fbf-af3d-49e2-8a7e-0848b39fc181\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srmtb"
Oct 31 00:19:16.247147 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:16.247075    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e3c9fbf-af3d-49e2-8a7e-0848b39fc181-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-srmtb\" (UID: \"6e3c9fbf-af3d-49e2-8a7e-0848b39fc181\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srmtb"
Oct 31 00:19:16.247727 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:16.247693    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e3c9fbf-af3d-49e2-8a7e-0848b39fc181-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-srmtb\" (UID: \"6e3c9fbf-af3d-49e2-8a7e-0848b39fc181\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srmtb"
Oct 31 00:19:16.249747 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:16.249723    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e3c9fbf-af3d-49e2-8a7e-0848b39fc181-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-srmtb\" (UID: \"6e3c9fbf-af3d-49e2-8a7e-0848b39fc181\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srmtb"
Oct 31 00:19:16.249865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:16.249782    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e3c9fbf-af3d-49e2-8a7e-0848b39fc181-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-srmtb\" (UID: \"6e3c9fbf-af3d-49e2-8a7e-0848b39fc181\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srmtb"
Oct 31 00:19:16.273945 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:16.273900    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-459sd\" (UniqueName: \"kubernetes.io/projected/6e3c9fbf-af3d-49e2-8a7e-0848b39fc181-kube-api-access-459sd\") pod \"prometheus-operator-9cd89b74c-srmtb\" (UID: \"6e3c9fbf-af3d-49e2-8a7e-0848b39fc181\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srmtb"
Oct 31 00:19:16.349462 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:16.349383    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-srmtb"
Oct 31 00:19:16.668957 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:16.668928    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-srmtb"]
Oct 31 00:19:16.669902 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:19:16.669872    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e3c9fbf_af3d_49e2_8a7e_0848b39fc181.slice/crio-4450c653b15cf11dae85dae64d24f3303b0d7488e132ebb4dbc5e0c9d7aadc44 WatchSource:0}: Error finding container 4450c653b15cf11dae85dae64d24f3303b0d7488e132ebb4dbc5e0c9d7aadc44: Status 404 returned error can't find the container with id 4450c653b15cf11dae85dae64d24f3303b0d7488e132ebb4dbc5e0c9d7aadc44
Oct 31 00:19:17.024872 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:17.024777    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-srmtb" event={"ID":"6e3c9fbf-af3d-49e2-8a7e-0848b39fc181","Type":"ContainerStarted","Data":"4450c653b15cf11dae85dae64d24f3303b0d7488e132ebb4dbc5e0c9d7aadc44"}
Oct 31 00:19:18.032896 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:18.032859    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-srmtb" event={"ID":"6e3c9fbf-af3d-49e2-8a7e-0848b39fc181","Type":"ContainerStarted","Data":"aa618e5a90189f101ad450530075a811c0ca49f8af0eb3b1b640de09dc6ebedb"}
Oct 31 00:19:19.039300 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:19.039263    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-srmtb" event={"ID":"6e3c9fbf-af3d-49e2-8a7e-0848b39fc181","Type":"ContainerStarted","Data":"126eaec233c3de3947264efd35e4d5831f2512bb55a97fff82f3ac36d8d82198"}
Oct 31 00:19:19.096374 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:19.096307    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-srmtb" podStartSLOduration=2.826649266 podStartE2EDuration="4.096289043s" totalImagesPullingTime="1.269639777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:19.095400898 +0000 UTC m=+19.317436439" watchObservedRunningTime="2026-10-31 00:19:19.096289043 +0000 UTC m=+19.318324586"
Oct 31 00:19:21.871319 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:21.871282    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-czprd"]
Oct 31 00:19:21.874748 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:21.874726    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:21.899937 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:21.899901    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 00:19:21.912404 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:21.912369    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pk56l\""
Oct 31 00:19:21.912576 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:21.912497    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 00:19:21.992047 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:21.992008    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9tnd\" (UniqueName: \"kubernetes.io/projected/4a336e35-d2f5-4fac-9ac2-6784b9b34108-kube-api-access-c9tnd\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:21.992047 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:21.992053    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4a336e35-d2f5-4fac-9ac2-6784b9b34108-node-exporter-textfile\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:21.992308 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:21.992078    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a336e35-d2f5-4fac-9ac2-6784b9b34108-node-exporter-wtmp\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:21.992308 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:21.992243    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a336e35-d2f5-4fac-9ac2-6784b9b34108-node-exporter-tls\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:21.992308 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:21.992293    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a336e35-d2f5-4fac-9ac2-6784b9b34108-root\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:21.992465 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:21.992322    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a336e35-d2f5-4fac-9ac2-6784b9b34108-metrics-client-ca\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:21.992465 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:21.992390    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a336e35-d2f5-4fac-9ac2-6784b9b34108-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:21.992465 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:21.992452    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a336e35-d2f5-4fac-9ac2-6784b9b34108-sys\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:21.992617 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:21.992501    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4a336e35-d2f5-4fac-9ac2-6784b9b34108-node-exporter-accelerators-collector-config\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:21.996789 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:21.996767    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 00:19:22.093817 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:22.093781    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9tnd\" (UniqueName: \"kubernetes.io/projected/4a336e35-d2f5-4fac-9ac2-6784b9b34108-kube-api-access-c9tnd\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:22.093817 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:22.093819    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4a336e35-d2f5-4fac-9ac2-6784b9b34108-node-exporter-textfile\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:22.094062 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:22.093840    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a336e35-d2f5-4fac-9ac2-6784b9b34108-node-exporter-wtmp\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:22.094062 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:22.093860    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a336e35-d2f5-4fac-9ac2-6784b9b34108-node-exporter-tls\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:22.094062 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:22.093893    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a336e35-d2f5-4fac-9ac2-6784b9b34108-root\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:22.094062 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:22.093924    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a336e35-d2f5-4fac-9ac2-6784b9b34108-metrics-client-ca\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:22.094062 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:22.093967    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a336e35-d2f5-4fac-9ac2-6784b9b34108-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:22.094062 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:22.093998    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a336e35-d2f5-4fac-9ac2-6784b9b34108-sys\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:22.094062 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:22.094014    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4a336e35-d2f5-4fac-9ac2-6784b9b34108-node-exporter-accelerators-collector-config\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:22.094383 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:22.094239    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a336e35-d2f5-4fac-9ac2-6784b9b34108-root\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:22.094383 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:22.094256    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4a336e35-d2f5-4fac-9ac2-6784b9b34108-node-exporter-textfile\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:22.094383 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:22.094373    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a336e35-d2f5-4fac-9ac2-6784b9b34108-node-exporter-wtmp\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:22.094548 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:22.094530    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4a336e35-d2f5-4fac-9ac2-6784b9b34108-node-exporter-accelerators-collector-config\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:22.094610 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:22.094602    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a336e35-d2f5-4fac-9ac2-6784b9b34108-sys\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:22.095046 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:22.095023    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a336e35-d2f5-4fac-9ac2-6784b9b34108-metrics-client-ca\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:22.096770 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:22.096747    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a336e35-d2f5-4fac-9ac2-6784b9b34108-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:22.096885 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:22.096866    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a336e35-d2f5-4fac-9ac2-6784b9b34108-node-exporter-tls\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:22.119508 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:22.119439    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9tnd\" (UniqueName: \"kubernetes.io/projected/4a336e35-d2f5-4fac-9ac2-6784b9b34108-kube-api-access-c9tnd\") pod \"node-exporter-czprd\" (UID: \"4a336e35-d2f5-4fac-9ac2-6784b9b34108\") " pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:22.192361 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:22.192266    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-czprd"
Oct 31 00:19:22.206815 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:19:22.206780    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a336e35_d2f5_4fac_9ac2_6784b9b34108.slice/crio-a171e0817f1efd82cb5c9414cc65f2b0f9ead986b09c2ace2c1573319ab700ac WatchSource:0}: Error finding container a171e0817f1efd82cb5c9414cc65f2b0f9ead986b09c2ace2c1573319ab700ac: Status 404 returned error can't find the container with id a171e0817f1efd82cb5c9414cc65f2b0f9ead986b09c2ace2c1573319ab700ac
Oct 31 00:19:23.056220 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:23.056180    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-czprd" event={"ID":"4a336e35-d2f5-4fac-9ac2-6784b9b34108","Type":"ContainerStarted","Data":"a171e0817f1efd82cb5c9414cc65f2b0f9ead986b09c2ace2c1573319ab700ac"}
Oct 31 00:19:24.062122 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:24.062076    4327 generic.go:356] "Generic (PLEG): container finished" podID="4a336e35-d2f5-4fac-9ac2-6784b9b34108" containerID="b8625c99b5f02454669e921afa9c0a2d4a462952b1ea046318570a5277fbebee" exitCode=0
Oct 31 00:19:24.062572 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:24.062160    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-czprd" event={"ID":"4a336e35-d2f5-4fac-9ac2-6784b9b34108","Type":"ContainerDied","Data":"b8625c99b5f02454669e921afa9c0a2d4a462952b1ea046318570a5277fbebee"}
Oct 31 00:19:25.068752 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:25.068713    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-czprd" event={"ID":"4a336e35-d2f5-4fac-9ac2-6784b9b34108","Type":"ContainerStarted","Data":"98ea6e782622d6ebeaf02f6950d61e9432d7430d2bfbd18120bc21ff4108eb20"}
Oct 31 00:19:25.069193 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:25.068817    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-czprd" event={"ID":"4a336e35-d2f5-4fac-9ac2-6784b9b34108","Type":"ContainerStarted","Data":"f57f24166a10e9bda46fabe125a6a90ca9dbefe09bc90d731e741c611d0075dd"}
Oct 31 00:19:25.142834 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:25.142768    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-czprd" podStartSLOduration=2.7093072510000002 podStartE2EDuration="4.14275212s" totalImagesPullingTime="1.333786298s" totalInitContainerRuntime="99.658571ms" isStatefulPod=true podCreationTimestamp="2026-10-31 00:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:25.129548576 +0000 UTC m=+25.351584156" watchObservedRunningTime="2026-10-31 00:19:25.14275212 +0000 UTC m=+25.364787662"
Oct 31 00:19:26.938990 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:26.938955    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g24ff"
Oct 31 00:19:29.085159 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:29.085066    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-4k28w" event={"ID":"93aea69b-c34d-4a21-acf8-a71d62fe8ef2","Type":"ContainerStarted","Data":"6bf2ae1412c51ab56861eaf1196eaf6432ba6bd9e81a11ec8e5bb6a7359aa991"}
Oct 31 00:19:29.085599 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:29.085228    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-4k28w"
Oct 31 00:19:29.086739 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:29.086712    4327 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-4k28w container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.11:8080/\": dial tcp 10.133.0.11:8080: connect: connection refused" start-of-body=
Oct 31 00:19:29.086854 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:29.086773    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-4k28w" podUID="93aea69b-c34d-4a21-acf8-a71d62fe8ef2" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.11:8080/\": dial tcp 10.133.0.11:8080: connect: connection refused"
Oct 31 00:19:30.110277 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:30.110244    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-4k28w"
Oct 31 00:19:30.183089 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:30.183026    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-4k28w" podStartSLOduration=2.953240489 podStartE2EDuration="19.183010161s" totalImagesPullingTime="16.229769672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:29.168105347 +0000 UTC m=+29.390140900" watchObservedRunningTime="2026-10-31 00:19:30.183010161 +0000 UTC m=+30.405045703"
Oct 31 00:19:31.962857 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:31.962821    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vdxch"
Oct 31 00:19:34.015333 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:34.015302    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-564986b4bd-526b7"
Oct 31 00:19:43.100651 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:43.100520    4327 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g24ff_17401e80-fe41-417f-9840-acd7a68d17e3/konnectivity-agent/0.log" limit=1024
Oct 31 00:19:43.391926 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:43.391831    4327 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g24ff_17401e80-fe41-417f-9840-acd7a68d17e3/konnectivity-agent/0.log" limit=1024
Oct 31 00:19:45.292436 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:19:45.292398    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:19:58.660603 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.660564    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-f758cf87d-dh7dw"]
Oct 31 00:19:58.694828 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.694792    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-f758cf87d-dh7dw"]
Oct 31 00:19:58.694992 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.694906    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw"
Oct 31 00:19:58.718966 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.711523    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 31 00:19:58.718966 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.711524    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 31 00:19:58.718966 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.711521    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 31 00:19:58.718966 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.711808    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-vk562\""
Oct 31 00:19:58.718966 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.711983    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 31 00:19:58.724277 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.724253    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 31 00:19:58.738005 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.737974    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 31 00:19:58.766542 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.766506    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b0c912b9-caee-4949-9010-5cbe2a5664c5-telemeter-client-tls\") pod \"telemeter-client-f758cf87d-dh7dw\" (UID: \"b0c912b9-caee-4949-9010-5cbe2a5664c5\") " pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw"
Oct 31 00:19:58.766542 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.766545    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b0c912b9-caee-4949-9010-5cbe2a5664c5-secret-telemeter-client\") pod \"telemeter-client-f758cf87d-dh7dw\" (UID: \"b0c912b9-caee-4949-9010-5cbe2a5664c5\") " pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw"
Oct 31 00:19:58.766783 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.766567    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdhjd\" (UniqueName: \"kubernetes.io/projected/b0c912b9-caee-4949-9010-5cbe2a5664c5-kube-api-access-tdhjd\") pod \"telemeter-client-f758cf87d-dh7dw\" (UID: \"b0c912b9-caee-4949-9010-5cbe2a5664c5\") " pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw"
Oct 31 00:19:58.766783 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.766673    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0c912b9-caee-4949-9010-5cbe2a5664c5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-f758cf87d-dh7dw\" (UID: \"b0c912b9-caee-4949-9010-5cbe2a5664c5\") " pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw"
Oct 31 00:19:58.766783 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.766744    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0c912b9-caee-4949-9010-5cbe2a5664c5-metrics-client-ca\") pod \"telemeter-client-f758cf87d-dh7dw\" (UID: \"b0c912b9-caee-4949-9010-5cbe2a5664c5\") " pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw"
Oct 31 00:19:58.766880 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.766793    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b0c912b9-caee-4949-9010-5cbe2a5664c5-federate-client-tls\") pod \"telemeter-client-f758cf87d-dh7dw\" (UID: \"b0c912b9-caee-4949-9010-5cbe2a5664c5\") " pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw"
Oct 31 00:19:58.766880 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.766823    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0c912b9-caee-4949-9010-5cbe2a5664c5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-f758cf87d-dh7dw\" (UID: \"b0c912b9-caee-4949-9010-5cbe2a5664c5\") " pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw"
Oct 31 00:19:58.766880 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.766842    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0c912b9-caee-4949-9010-5cbe2a5664c5-serving-certs-ca-bundle\") pod \"telemeter-client-f758cf87d-dh7dw\" (UID: \"b0c912b9-caee-4949-9010-5cbe2a5664c5\") " pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw"
Oct 31 00:19:58.867391 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.867355    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b0c912b9-caee-4949-9010-5cbe2a5664c5-telemeter-client-tls\") pod \"telemeter-client-f758cf87d-dh7dw\" (UID: \"b0c912b9-caee-4949-9010-5cbe2a5664c5\") " pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw"
Oct 31 00:19:58.867391 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.867393    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b0c912b9-caee-4949-9010-5cbe2a5664c5-secret-telemeter-client\") pod \"telemeter-client-f758cf87d-dh7dw\" (UID: \"b0c912b9-caee-4949-9010-5cbe2a5664c5\") " pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw"
Oct 31 00:19:58.867618 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.867416    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdhjd\" (UniqueName: \"kubernetes.io/projected/b0c912b9-caee-4949-9010-5cbe2a5664c5-kube-api-access-tdhjd\") pod \"telemeter-client-f758cf87d-dh7dw\" (UID: \"b0c912b9-caee-4949-9010-5cbe2a5664c5\") " pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw"
Oct 31 00:19:58.867618 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.867447    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0c912b9-caee-4949-9010-5cbe2a5664c5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-f758cf87d-dh7dw\" (UID: \"b0c912b9-caee-4949-9010-5cbe2a5664c5\") " pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw"
Oct 31 00:19:58.867618 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.867475    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0c912b9-caee-4949-9010-5cbe2a5664c5-metrics-client-ca\") pod \"telemeter-client-f758cf87d-dh7dw\" (UID: \"b0c912b9-caee-4949-9010-5cbe2a5664c5\") " pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw"
Oct 31 00:19:58.867618 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.867523    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b0c912b9-caee-4949-9010-5cbe2a5664c5-federate-client-tls\") pod \"telemeter-client-f758cf87d-dh7dw\" (UID: \"b0c912b9-caee-4949-9010-5cbe2a5664c5\") " pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw"
Oct 31 00:19:58.867618 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.867565    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0c912b9-caee-4949-9010-5cbe2a5664c5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-f758cf87d-dh7dw\" (UID: \"b0c912b9-caee-4949-9010-5cbe2a5664c5\") " pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw"
Oct 31 00:19:58.867618 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.867593    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0c912b9-caee-4949-9010-5cbe2a5664c5-serving-certs-ca-bundle\") pod \"telemeter-client-f758cf87d-dh7dw\" (UID: \"b0c912b9-caee-4949-9010-5cbe2a5664c5\") " pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw"
Oct 31 00:19:58.868452 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.868387    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0c912b9-caee-4949-9010-5cbe2a5664c5-metrics-client-ca\") pod \"telemeter-client-f758cf87d-dh7dw\" (UID: \"b0c912b9-caee-4949-9010-5cbe2a5664c5\") " pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw"
Oct 31 00:19:58.868452 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.868436    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0c912b9-caee-4949-9010-5cbe2a5664c5-serving-certs-ca-bundle\") pod \"telemeter-client-f758cf87d-dh7dw\" (UID: \"b0c912b9-caee-4949-9010-5cbe2a5664c5\") " pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw"
Oct 31 00:19:58.868808 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.868782    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0c912b9-caee-4949-9010-5cbe2a5664c5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-f758cf87d-dh7dw\" (UID: \"b0c912b9-caee-4949-9010-5cbe2a5664c5\") " pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw"
Oct 31 00:19:58.870121 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.870088    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b0c912b9-caee-4949-9010-5cbe2a5664c5-telemeter-client-tls\") pod \"telemeter-client-f758cf87d-dh7dw\" (UID: \"b0c912b9-caee-4949-9010-5cbe2a5664c5\") " pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw"
Oct 31 00:19:58.870209 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.870143    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0c912b9-caee-4949-9010-5cbe2a5664c5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-f758cf87d-dh7dw\" (UID: \"b0c912b9-caee-4949-9010-5cbe2a5664c5\") " pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw"
Oct 31 00:19:58.870209 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.870201    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b0c912b9-caee-4949-9010-5cbe2a5664c5-secret-telemeter-client\") pod \"telemeter-client-f758cf87d-dh7dw\" (UID: \"b0c912b9-caee-4949-9010-5cbe2a5664c5\") " pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw"
Oct 31 00:19:58.870281 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.870261    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b0c912b9-caee-4949-9010-5cbe2a5664c5-federate-client-tls\") pod \"telemeter-client-f758cf87d-dh7dw\" (UID: \"b0c912b9-caee-4949-9010-5cbe2a5664c5\") " pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw"
Oct 31 00:19:58.891405 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:58.891369    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdhjd\" (UniqueName: \"kubernetes.io/projected/b0c912b9-caee-4949-9010-5cbe2a5664c5-kube-api-access-tdhjd\") pod \"telemeter-client-f758cf87d-dh7dw\" (UID: \"b0c912b9-caee-4949-9010-5cbe2a5664c5\") " pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw"
Oct 31 00:19:59.051484 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:59.051362    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw"
Oct 31 00:19:59.424926 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:19:59.424884    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-f758cf87d-dh7dw"]
Oct 31 00:19:59.426037 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:19:59.426008    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c912b9_caee_4949_9010_5cbe2a5664c5.slice/crio-aecf0c8c1e23a08e035831798595199f7f186c7567ce97f21cf63742c21a3777 WatchSource:0}: Error finding container aecf0c8c1e23a08e035831798595199f7f186c7567ce97f21cf63742c21a3777: Status 404 returned error can't find the container with id aecf0c8c1e23a08e035831798595199f7f186c7567ce97f21cf63742c21a3777
Oct 31 00:20:00.183701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:20:00.183650    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw" event={"ID":"b0c912b9-caee-4949-9010-5cbe2a5664c5","Type":"ContainerStarted","Data":"aecf0c8c1e23a08e035831798595199f7f186c7567ce97f21cf63742c21a3777"}
Oct 31 00:20:02.192029 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:20:02.191992    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw" event={"ID":"b0c912b9-caee-4949-9010-5cbe2a5664c5","Type":"ContainerStarted","Data":"759a1f4d94591b09f239dfa5a506267657f8dc07f211e5b7dd6a49251710e020"}
Oct 31 00:20:03.159989 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:20:03.159954    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h26l7_8072caf6-1730-4100-a87a-630ace22a370/serve-healthcheck-canary/0.log"
Oct 31 00:20:04.199816 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:20:04.199778    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw" event={"ID":"b0c912b9-caee-4949-9010-5cbe2a5664c5","Type":"ContainerStarted","Data":"959af27f77247775afa996a87692b7fe05aa1777fcbbf938b38d77f267355be3"}
Oct 31 00:20:04.200235 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:20:04.199847    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw" event={"ID":"b0c912b9-caee-4949-9010-5cbe2a5664c5","Type":"ContainerStarted","Data":"da1dae4b8d35e5a967725c5145368c6c32a1071b2f75e9370477934f6d561246"}
Oct 31 00:20:04.286361 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:20:04.286302    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-f758cf87d-dh7dw" podStartSLOduration=2.456161477 podStartE2EDuration="6.286288633s" totalImagesPullingTime="3.830127156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:04.2734015 +0000 UTC m=+64.495437042" watchObservedRunningTime="2026-10-31 00:20:04.286288633 +0000 UTC m=+64.508324175"
Oct 31 00:20:15.291774 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:20:15.291735    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:20:45.291415 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:20:45.291374    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:21:15.291712 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:21:15.291602    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:21:45.290134 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:21:45.290088    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:22:05.148325 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.148289    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27"]
Oct 31 00:22:05.150406 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.150391    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27"
Oct 31 00:22:05.171096 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.171071    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-l9j5t\""
Oct 31 00:22:05.183927 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.183903    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 00:22:05.183927 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.183905    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 00:22:05.188785 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.188764    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27"]
Oct 31 00:22:05.219883 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.219851    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d"]
Oct 31 00:22:05.222050 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.222035    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d"
Oct 31 00:22:05.245007 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.244979    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d"]
Oct 31 00:22:05.256292 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.256266    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-85jgg\""
Oct 31 00:22:05.274145 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.274116    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1baf842-4fab-4dcf-9923-e9ce06f6ad40-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27\" (UID: \"d1baf842-4fab-4dcf-9923-e9ce06f6ad40\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27"
Oct 31 00:22:05.274277 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.274183    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jctdn\" (UniqueName: \"kubernetes.io/projected/d1baf842-4fab-4dcf-9923-e9ce06f6ad40-kube-api-access-jctdn\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27\" (UID: \"d1baf842-4fab-4dcf-9923-e9ce06f6ad40\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27"
Oct 31 00:22:05.274340 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.274298    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1baf842-4fab-4dcf-9923-e9ce06f6ad40-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27\" (UID: \"d1baf842-4fab-4dcf-9923-e9ce06f6ad40\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27"
Oct 31 00:22:05.374720 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.374672    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1baf842-4fab-4dcf-9923-e9ce06f6ad40-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27\" (UID: \"d1baf842-4fab-4dcf-9923-e9ce06f6ad40\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27"
Oct 31 00:22:05.374890 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.374740    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb545ea6-d001-44ac-b76e-530b6a1d1aad-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d\" (UID: \"eb545ea6-d001-44ac-b76e-530b6a1d1aad\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d"
Oct 31 00:22:05.374890 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.374770    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1baf842-4fab-4dcf-9923-e9ce06f6ad40-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27\" (UID: \"d1baf842-4fab-4dcf-9923-e9ce06f6ad40\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27"
Oct 31 00:22:05.374962 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.374933    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb545ea6-d001-44ac-b76e-530b6a1d1aad-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d\" (UID: \"eb545ea6-d001-44ac-b76e-530b6a1d1aad\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d"
Oct 31 00:22:05.374994 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.374971    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jctdn\" (UniqueName: \"kubernetes.io/projected/d1baf842-4fab-4dcf-9923-e9ce06f6ad40-kube-api-access-jctdn\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27\" (UID: \"d1baf842-4fab-4dcf-9923-e9ce06f6ad40\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27"
Oct 31 00:22:05.375029 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.375000    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t746\" (UniqueName: \"kubernetes.io/projected/eb545ea6-d001-44ac-b76e-530b6a1d1aad-kube-api-access-2t746\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d\" (UID: \"eb545ea6-d001-44ac-b76e-530b6a1d1aad\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d"
Oct 31 00:22:05.375029 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.375010    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1baf842-4fab-4dcf-9923-e9ce06f6ad40-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27\" (UID: \"d1baf842-4fab-4dcf-9923-e9ce06f6ad40\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27"
Oct 31 00:22:05.375094 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.375066    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1baf842-4fab-4dcf-9923-e9ce06f6ad40-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27\" (UID: \"d1baf842-4fab-4dcf-9923-e9ce06f6ad40\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27"
Oct 31 00:22:05.404227 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.404166    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jctdn\" (UniqueName: \"kubernetes.io/projected/d1baf842-4fab-4dcf-9923-e9ce06f6ad40-kube-api-access-jctdn\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27\" (UID: \"d1baf842-4fab-4dcf-9923-e9ce06f6ad40\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27"
Oct 31 00:22:05.466364 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.466330    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27"
Oct 31 00:22:05.475428 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.475383    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb545ea6-d001-44ac-b76e-530b6a1d1aad-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d\" (UID: \"eb545ea6-d001-44ac-b76e-530b6a1d1aad\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d"
Oct 31 00:22:05.475508 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.475419    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t746\" (UniqueName: \"kubernetes.io/projected/eb545ea6-d001-44ac-b76e-530b6a1d1aad-kube-api-access-2t746\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d\" (UID: \"eb545ea6-d001-44ac-b76e-530b6a1d1aad\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d"
Oct 31 00:22:05.475508 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.475484    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb545ea6-d001-44ac-b76e-530b6a1d1aad-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d\" (UID: \"eb545ea6-d001-44ac-b76e-530b6a1d1aad\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d"
Oct 31 00:22:05.475795 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.475765    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb545ea6-d001-44ac-b76e-530b6a1d1aad-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d\" (UID: \"eb545ea6-d001-44ac-b76e-530b6a1d1aad\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d"
Oct 31 00:22:05.475886 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.475871    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb545ea6-d001-44ac-b76e-530b6a1d1aad-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d\" (UID: \"eb545ea6-d001-44ac-b76e-530b6a1d1aad\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d"
Oct 31 00:22:05.476354 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.476336    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"]
Oct 31 00:22:05.478302 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.478280    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:22:05.494573 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.494538    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 00:22:05.494720 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.494706    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 00:22:05.498178 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.498159    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 00:22:05.498178 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.498169    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 00:22:05.498309 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.498203    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zgxxb\""
Oct 31 00:22:05.499262 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.499249    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 00:22:05.505830 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.505810    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t746\" (UniqueName: \"kubernetes.io/projected/eb545ea6-d001-44ac-b76e-530b6a1d1aad-kube-api-access-2t746\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d\" (UID: \"eb545ea6-d001-44ac-b76e-530b6a1d1aad\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d"
Oct 31 00:22:05.510574 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.510546    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"]
Oct 31 00:22:05.534137 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.533942    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d"
Oct 31 00:22:05.576707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.576594    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25b686d-53ec-43f2-984b-bb6723f8fb50-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-8dkw7\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:22:05.576707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.576645    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh5fv\" (UniqueName: \"kubernetes.io/projected/a25b686d-53ec-43f2-984b-bb6723f8fb50-kube-api-access-hh5fv\") pod \"istiod-openshift-gateway-7876bfcb74-8dkw7\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:22:05.576707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.576669    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-8dkw7\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:22:05.576995 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.576836    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-8dkw7\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:22:05.576995 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.576928    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a25b686d-53ec-43f2-984b-bb6723f8fb50-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-8dkw7\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:22:05.577491 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.576958    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-8dkw7\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:22:05.577491 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.577190    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-8dkw7\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:22:05.595259 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.595234    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 00:22:05.678798 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.678710    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25b686d-53ec-43f2-984b-bb6723f8fb50-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-8dkw7\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:22:05.678798 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.678767    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh5fv\" (UniqueName: \"kubernetes.io/projected/a25b686d-53ec-43f2-984b-bb6723f8fb50-kube-api-access-hh5fv\") pod \"istiod-openshift-gateway-7876bfcb74-8dkw7\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:22:05.678798 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.678796    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-8dkw7\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:22:05.679051 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.678828    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-8dkw7\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:22:05.679104 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.679047    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a25b686d-53ec-43f2-984b-bb6723f8fb50-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-8dkw7\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:22:05.679104 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.679084    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-8dkw7\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:22:05.679198 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.679112    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-8dkw7\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:22:05.679670 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.679646    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-8dkw7\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:22:05.681128 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.681107    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-8dkw7\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:22:05.681206 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.681184    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a25b686d-53ec-43f2-984b-bb6723f8fb50-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-8dkw7\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:22:05.681412 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.681397    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25b686d-53ec-43f2-984b-bb6723f8fb50-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-8dkw7\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:22:05.681470 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.681454    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-8dkw7\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:22:05.702855 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.702831    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-8dkw7\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:22:05.703404 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.703387    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh5fv\" (UniqueName: \"kubernetes.io/projected/a25b686d-53ec-43f2-984b-bb6723f8fb50-kube-api-access-hh5fv\") pod \"istiod-openshift-gateway-7876bfcb74-8dkw7\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:22:05.761361 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.761333    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27"]
Oct 31 00:22:05.763565 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:22:05.763543    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1baf842_4fab_4dcf_9923_e9ce06f6ad40.slice/crio-02f46369d9ec684595011e9875d567b085d3ddffbd9d1dfed60a51b9fec47ccd WatchSource:0}: Error finding container 02f46369d9ec684595011e9875d567b085d3ddffbd9d1dfed60a51b9fec47ccd: Status 404 returned error can't find the container with id 02f46369d9ec684595011e9875d567b085d3ddffbd9d1dfed60a51b9fec47ccd
Oct 31 00:22:05.795897 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.795868    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:22:05.829436 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:05.829410    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d"]
Oct 31 00:22:05.830188 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:22:05.830160    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb545ea6_d001_44ac_b76e_530b6a1d1aad.slice/crio-d08d64dbefaafddd0007903dcf5084fee4d5f4f3b9dc96eeeb1f3f09ee8906f3 WatchSource:0}: Error finding container d08d64dbefaafddd0007903dcf5084fee4d5f4f3b9dc96eeeb1f3f09ee8906f3: Status 404 returned error can't find the container with id d08d64dbefaafddd0007903dcf5084fee4d5f4f3b9dc96eeeb1f3f09ee8906f3
Oct 31 00:22:06.099700 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:06.099529    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"]
Oct 31 00:22:06.100734 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:22:06.100707    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda25b686d_53ec_43f2_984b_bb6723f8fb50.slice/crio-a69aa733c43a90f5360961078e653bfdc20dd4ae3417cc72f00da79895f92d74 WatchSource:0}: Error finding container a69aa733c43a90f5360961078e653bfdc20dd4ae3417cc72f00da79895f92d74: Status 404 returned error can't find the container with id a69aa733c43a90f5360961078e653bfdc20dd4ae3417cc72f00da79895f92d74
Oct 31 00:22:06.593235 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:06.593190    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7" event={"ID":"a25b686d-53ec-43f2-984b-bb6723f8fb50","Type":"ContainerStarted","Data":"a69aa733c43a90f5360961078e653bfdc20dd4ae3417cc72f00da79895f92d74"}
Oct 31 00:22:06.594703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:06.594657    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d" event={"ID":"eb545ea6-d001-44ac-b76e-530b6a1d1aad","Type":"ContainerStarted","Data":"d08d64dbefaafddd0007903dcf5084fee4d5f4f3b9dc96eeeb1f3f09ee8906f3"}
Oct 31 00:22:06.596108 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:06.596074    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27" event={"ID":"d1baf842-4fab-4dcf-9923-e9ce06f6ad40","Type":"ContainerStarted","Data":"02f46369d9ec684595011e9875d567b085d3ddffbd9d1dfed60a51b9fec47ccd"}
Oct 31 00:22:09.047011 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:09.046934    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:22:09.047478 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:09.047020    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:22:09.616185 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:09.615578    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7" event={"ID":"a25b686d-53ec-43f2-984b-bb6723f8fb50","Type":"ContainerStarted","Data":"125f7f6fda7f9ec0fe76bd2a7c6d2800279dc3bb4641eb377d40d8eb6a5ae5df"}
Oct 31 00:22:09.616185 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:09.615862    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:22:09.668485 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:09.668279    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7" podStartSLOduration=1.725108696 podStartE2EDuration="4.668258429s" totalImagesPullingTime="2.943149733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:09.666961481 +0000 UTC m=+189.888997025" watchObservedRunningTime="2026-10-31 00:22:09.668258429 +0000 UTC m=+189.890293974"
Oct 31 00:22:10.621744 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:10.621707    4327 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-8dkw7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:22:10.622160 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:10.621777    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7" podUID="a25b686d-53ec-43f2-984b-bb6723f8fb50" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:22:13.621048 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:13.621010    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:22:13.632445 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:13.632419    4327 generic.go:356] "Generic (PLEG): container finished" podID="d1baf842-4fab-4dcf-9923-e9ce06f6ad40" containerID="c3b34577e086b000d2df83effe2e392ac7d457433a4c6ea687dfb5c2697c85c1" exitCode=0
Oct 31 00:22:13.632596 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:13.632502    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27" event={"ID":"d1baf842-4fab-4dcf-9923-e9ce06f6ad40","Type":"ContainerDied","Data":"c3b34577e086b000d2df83effe2e392ac7d457433a4c6ea687dfb5c2697c85c1"}
Oct 31 00:22:13.633949 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:13.633926    4327 generic.go:356] "Generic (PLEG): container finished" podID="eb545ea6-d001-44ac-b76e-530b6a1d1aad" containerID="7f82867431e8187a9debb3e46c0265a0ff2f5fdb5e034c4e4ad17e2481c5a750" exitCode=0
Oct 31 00:22:13.634053 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:13.633956    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d" event={"ID":"eb545ea6-d001-44ac-b76e-530b6a1d1aad","Type":"ContainerDied","Data":"7f82867431e8187a9debb3e46c0265a0ff2f5fdb5e034c4e4ad17e2481c5a750"}
Oct 31 00:22:14.640862 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:14.640828    4327 generic.go:356] "Generic (PLEG): container finished" podID="eb545ea6-d001-44ac-b76e-530b6a1d1aad" containerID="46a330cbae73d2704592f5f853c3048e5c69b2123e576acdd6f6e0de3ad2bc01" exitCode=0
Oct 31 00:22:14.641345 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:14.640907    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d" event={"ID":"eb545ea6-d001-44ac-b76e-530b6a1d1aad","Type":"ContainerDied","Data":"46a330cbae73d2704592f5f853c3048e5c69b2123e576acdd6f6e0de3ad2bc01"}
Oct 31 00:22:15.297938 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:22:15.297861    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:22:16.651783 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:16.651740    4327 generic.go:356] "Generic (PLEG): container finished" podID="d1baf842-4fab-4dcf-9923-e9ce06f6ad40" containerID="095630eeb3a273325e4d7ad0c4b989b803baaa2869f313c70b6f2a1b9c638bf2" exitCode=0
Oct 31 00:22:16.652313 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:16.651837    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27" event={"ID":"d1baf842-4fab-4dcf-9923-e9ce06f6ad40","Type":"ContainerDied","Data":"095630eeb3a273325e4d7ad0c4b989b803baaa2869f313c70b6f2a1b9c638bf2"}
Oct 31 00:22:21.674499 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:21.674459    4327 generic.go:356] "Generic (PLEG): container finished" podID="eb545ea6-d001-44ac-b76e-530b6a1d1aad" containerID="4b0dec83233b21c7708e1749cf7e6eec088919691c3f73b405cbdad1b0a88cfe" exitCode=0
Oct 31 00:22:21.674944 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:21.674546    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d" event={"ID":"eb545ea6-d001-44ac-b76e-530b6a1d1aad","Type":"ContainerDied","Data":"4b0dec83233b21c7708e1749cf7e6eec088919691c3f73b405cbdad1b0a88cfe"}
Oct 31 00:22:22.819792 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:22.819756    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d"
Oct 31 00:22:22.840387 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:22.840362    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb545ea6-d001-44ac-b76e-530b6a1d1aad-util\" (UniqueName: \"kubernetes.io/empty-dir/eb545ea6-d001-44ac-b76e-530b6a1d1aad-util\") pod \"eb545ea6-d001-44ac-b76e-530b6a1d1aad\" (UID: \"eb545ea6-d001-44ac-b76e-530b6a1d1aad\") "
Oct 31 00:22:22.840533 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:22.840421    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb545ea6-d001-44ac-b76e-530b6a1d1aad-bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb545ea6-d001-44ac-b76e-530b6a1d1aad-bundle\") pod \"eb545ea6-d001-44ac-b76e-530b6a1d1aad\" (UID: \"eb545ea6-d001-44ac-b76e-530b6a1d1aad\") "
Oct 31 00:22:22.840575 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:22.840532    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb545ea6-d001-44ac-b76e-530b6a1d1aad-kube-api-access-2t746\" (UniqueName: \"kubernetes.io/projected/eb545ea6-d001-44ac-b76e-530b6a1d1aad-kube-api-access-2t746\") pod \"eb545ea6-d001-44ac-b76e-530b6a1d1aad\" (UID: \"eb545ea6-d001-44ac-b76e-530b6a1d1aad\") "
Oct 31 00:22:22.840960 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:22.840935    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb545ea6-d001-44ac-b76e-530b6a1d1aad-bundle" pod "eb545ea6-d001-44ac-b76e-530b6a1d1aad" (UID: "eb545ea6-d001-44ac-b76e-530b6a1d1aad"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:22.842657 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:22.842628    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb545ea6-d001-44ac-b76e-530b6a1d1aad-kube-api-access-2t746" pod "eb545ea6-d001-44ac-b76e-530b6a1d1aad" (UID: "eb545ea6-d001-44ac-b76e-530b6a1d1aad"). InnerVolumeSpecName "kube-api-access-2t746". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:22:22.844793 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:22.844758    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb545ea6-d001-44ac-b76e-530b6a1d1aad-util" pod "eb545ea6-d001-44ac-b76e-530b6a1d1aad" (UID: "eb545ea6-d001-44ac-b76e-530b6a1d1aad"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:22.941524 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:22.941442    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t746\" (UniqueName: \"kubernetes.io/projected/eb545ea6-d001-44ac-b76e-530b6a1d1aad-kube-api-access-2t746\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:22:22.941524 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:22.941481    4327 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb545ea6-d001-44ac-b76e-530b6a1d1aad-util\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:22:22.941524 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:22.941496    4327 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb545ea6-d001-44ac-b76e-530b6a1d1aad-bundle\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:22:23.682979 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:23.682950    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d"
Oct 31 00:22:23.683159 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:23.682946    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wxj4d" event={"ID":"eb545ea6-d001-44ac-b76e-530b6a1d1aad","Type":"ContainerDied","Data":"d08d64dbefaafddd0007903dcf5084fee4d5f4f3b9dc96eeeb1f3f09ee8906f3"}
Oct 31 00:22:23.683159 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:23.683067    4327 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d08d64dbefaafddd0007903dcf5084fee4d5f4f3b9dc96eeeb1f3f09ee8906f3"
Oct 31 00:22:23.685038 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:23.685013    4327 generic.go:356] "Generic (PLEG): container finished" podID="d1baf842-4fab-4dcf-9923-e9ce06f6ad40" containerID="ca85b208de115a31db504c1eca16df307472ebaa20a7c670aae86e559f26625d" exitCode=0
Oct 31 00:22:23.685165 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:23.685047    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27" event={"ID":"d1baf842-4fab-4dcf-9923-e9ce06f6ad40","Type":"ContainerDied","Data":"ca85b208de115a31db504c1eca16df307472ebaa20a7c670aae86e559f26625d"}
Oct 31 00:22:24.810926 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:24.810900    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27"
Oct 31 00:22:24.857285 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:24.857259    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1baf842-4fab-4dcf-9923-e9ce06f6ad40-util\" (UniqueName: \"kubernetes.io/empty-dir/d1baf842-4fab-4dcf-9923-e9ce06f6ad40-util\") pod \"d1baf842-4fab-4dcf-9923-e9ce06f6ad40\" (UID: \"d1baf842-4fab-4dcf-9923-e9ce06f6ad40\") "
Oct 31 00:22:24.857442 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:24.857324    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1baf842-4fab-4dcf-9923-e9ce06f6ad40-kube-api-access-jctdn\" (UniqueName: \"kubernetes.io/projected/d1baf842-4fab-4dcf-9923-e9ce06f6ad40-kube-api-access-jctdn\") pod \"d1baf842-4fab-4dcf-9923-e9ce06f6ad40\" (UID: \"d1baf842-4fab-4dcf-9923-e9ce06f6ad40\") "
Oct 31 00:22:24.857442 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:24.857370    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1baf842-4fab-4dcf-9923-e9ce06f6ad40-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1baf842-4fab-4dcf-9923-e9ce06f6ad40-bundle\") pod \"d1baf842-4fab-4dcf-9923-e9ce06f6ad40\" (UID: \"d1baf842-4fab-4dcf-9923-e9ce06f6ad40\") "
Oct 31 00:22:24.858178 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:24.858149    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1baf842-4fab-4dcf-9923-e9ce06f6ad40-bundle" pod "d1baf842-4fab-4dcf-9923-e9ce06f6ad40" (UID: "d1baf842-4fab-4dcf-9923-e9ce06f6ad40"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:24.859648 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:24.859623    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1baf842-4fab-4dcf-9923-e9ce06f6ad40-kube-api-access-jctdn" pod "d1baf842-4fab-4dcf-9923-e9ce06f6ad40" (UID: "d1baf842-4fab-4dcf-9923-e9ce06f6ad40"). InnerVolumeSpecName "kube-api-access-jctdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:22:24.865380 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:24.865352    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1baf842-4fab-4dcf-9923-e9ce06f6ad40-util" pod "d1baf842-4fab-4dcf-9923-e9ce06f6ad40" (UID: "d1baf842-4fab-4dcf-9923-e9ce06f6ad40"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:24.958426 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:24.958357    4327 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1baf842-4fab-4dcf-9923-e9ce06f6ad40-util\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:22:24.958426 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:24.958387    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jctdn\" (UniqueName: \"kubernetes.io/projected/d1baf842-4fab-4dcf-9923-e9ce06f6ad40-kube-api-access-jctdn\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:22:24.958426 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:24.958397    4327 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1baf842-4fab-4dcf-9923-e9ce06f6ad40-bundle\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:22:25.700317 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:25.700283    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27" event={"ID":"d1baf842-4fab-4dcf-9923-e9ce06f6ad40","Type":"ContainerDied","Data":"02f46369d9ec684595011e9875d567b085d3ddffbd9d1dfed60a51b9fec47ccd"}
Oct 31 00:22:25.700486 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:25.700357    4327 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02f46369d9ec684595011e9875d567b085d3ddffbd9d1dfed60a51b9fec47ccd"
Oct 31 00:22:25.700486 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:25.700292    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c47s27"
Oct 31 00:22:39.653277 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.653237    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j"]
Oct 31 00:22:39.653924 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.653614    4327 cpu_manager.go:414] "containerMap: removing container" podUID="d1baf842-4fab-4dcf-9923-e9ce06f6ad40" containerName="extract"
Oct 31 00:22:39.653924 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.653631    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1baf842-4fab-4dcf-9923-e9ce06f6ad40" containerName="extract"
Oct 31 00:22:39.653924 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.653658    4327 cpu_manager.go:414] "containerMap: removing container" podUID="d1baf842-4fab-4dcf-9923-e9ce06f6ad40" containerName="util"
Oct 31 00:22:39.653924 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.653665    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1baf842-4fab-4dcf-9923-e9ce06f6ad40" containerName="util"
Oct 31 00:22:39.653924 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.653692    4327 cpu_manager.go:414] "containerMap: removing container" podUID="eb545ea6-d001-44ac-b76e-530b6a1d1aad" containerName="pull"
Oct 31 00:22:39.653924 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.653701    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb545ea6-d001-44ac-b76e-530b6a1d1aad" containerName="pull"
Oct 31 00:22:39.653924 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.653724    4327 cpu_manager.go:414] "containerMap: removing container" podUID="eb545ea6-d001-44ac-b76e-530b6a1d1aad" containerName="util"
Oct 31 00:22:39.653924 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.653731    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb545ea6-d001-44ac-b76e-530b6a1d1aad" containerName="util"
Oct 31 00:22:39.653924 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.653741    4327 cpu_manager.go:414] "containerMap: removing container" podUID="d1baf842-4fab-4dcf-9923-e9ce06f6ad40" containerName="pull"
Oct 31 00:22:39.653924 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.653748    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1baf842-4fab-4dcf-9923-e9ce06f6ad40" containerName="pull"
Oct 31 00:22:39.653924 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.653757    4327 cpu_manager.go:414] "containerMap: removing container" podUID="eb545ea6-d001-44ac-b76e-530b6a1d1aad" containerName="extract"
Oct 31 00:22:39.653924 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.653763    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb545ea6-d001-44ac-b76e-530b6a1d1aad" containerName="extract"
Oct 31 00:22:39.653924 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.653839    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb545ea6-d001-44ac-b76e-530b6a1d1aad" containerName="extract"
Oct 31 00:22:39.653924 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.653849    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1baf842-4fab-4dcf-9923-e9ce06f6ad40" containerName="extract"
Oct 31 00:22:39.658106 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.658085    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j"
Oct 31 00:22:39.731897 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.731862    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 00:22:39.732392 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.732376    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 00:22:39.732657 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.732643    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-2v5hv\""
Oct 31 00:22:39.732757 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.732653    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 00:22:39.732757 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.732658    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 00:22:39.734057 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.734037    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 00:22:39.765751 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.765722    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/df7eb9de-4a8d-41bc-be97-0ac12ae9f63c-metrics-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-nxt2j\" (UID: \"df7eb9de-4a8d-41bc-be97-0ac12ae9f63c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j"
Oct 31 00:22:39.765914 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.765782    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/df7eb9de-4a8d-41bc-be97-0ac12ae9f63c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-nxt2j\" (UID: \"df7eb9de-4a8d-41bc-be97-0ac12ae9f63c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j"
Oct 31 00:22:39.765914 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.765882    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vbh5\" (UniqueName: \"kubernetes.io/projected/df7eb9de-4a8d-41bc-be97-0ac12ae9f63c-kube-api-access-5vbh5\") pod \"opendatahub-operator-controller-manager-f465f7bb6-nxt2j\" (UID: \"df7eb9de-4a8d-41bc-be97-0ac12ae9f63c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j"
Oct 31 00:22:39.765998 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.765910    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df7eb9de-4a8d-41bc-be97-0ac12ae9f63c-webhook-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-nxt2j\" (UID: \"df7eb9de-4a8d-41bc-be97-0ac12ae9f63c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j"
Oct 31 00:22:39.765998 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.765935    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/df7eb9de-4a8d-41bc-be97-0ac12ae9f63c-manifests\") pod \"opendatahub-operator-controller-manager-f465f7bb6-nxt2j\" (UID: \"df7eb9de-4a8d-41bc-be97-0ac12ae9f63c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j"
Oct 31 00:22:39.765998 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.765983    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df7eb9de-4a8d-41bc-be97-0ac12ae9f63c-tmp\") pod \"opendatahub-operator-controller-manager-f465f7bb6-nxt2j\" (UID: \"df7eb9de-4a8d-41bc-be97-0ac12ae9f63c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j"
Oct 31 00:22:39.867320 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.867277    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vbh5\" (UniqueName: \"kubernetes.io/projected/df7eb9de-4a8d-41bc-be97-0ac12ae9f63c-kube-api-access-5vbh5\") pod \"opendatahub-operator-controller-manager-f465f7bb6-nxt2j\" (UID: \"df7eb9de-4a8d-41bc-be97-0ac12ae9f63c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j"
Oct 31 00:22:39.867512 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.867330    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df7eb9de-4a8d-41bc-be97-0ac12ae9f63c-webhook-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-nxt2j\" (UID: \"df7eb9de-4a8d-41bc-be97-0ac12ae9f63c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j"
Oct 31 00:22:39.867512 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.867351    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/df7eb9de-4a8d-41bc-be97-0ac12ae9f63c-manifests\") pod \"opendatahub-operator-controller-manager-f465f7bb6-nxt2j\" (UID: \"df7eb9de-4a8d-41bc-be97-0ac12ae9f63c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j"
Oct 31 00:22:39.867512 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.867367    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df7eb9de-4a8d-41bc-be97-0ac12ae9f63c-tmp\") pod \"opendatahub-operator-controller-manager-f465f7bb6-nxt2j\" (UID: \"df7eb9de-4a8d-41bc-be97-0ac12ae9f63c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j"
Oct 31 00:22:39.867512 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.867410    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/df7eb9de-4a8d-41bc-be97-0ac12ae9f63c-metrics-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-nxt2j\" (UID: \"df7eb9de-4a8d-41bc-be97-0ac12ae9f63c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j"
Oct 31 00:22:39.867512 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.867460    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/df7eb9de-4a8d-41bc-be97-0ac12ae9f63c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-nxt2j\" (UID: \"df7eb9de-4a8d-41bc-be97-0ac12ae9f63c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j"
Oct 31 00:22:39.868327 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.868293    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/df7eb9de-4a8d-41bc-be97-0ac12ae9f63c-manifests\") pod \"opendatahub-operator-controller-manager-f465f7bb6-nxt2j\" (UID: \"df7eb9de-4a8d-41bc-be97-0ac12ae9f63c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j"
Oct 31 00:22:39.872241 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.871104    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/df7eb9de-4a8d-41bc-be97-0ac12ae9f63c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-nxt2j\" (UID: \"df7eb9de-4a8d-41bc-be97-0ac12ae9f63c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j"
Oct 31 00:22:39.872241 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.871191    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df7eb9de-4a8d-41bc-be97-0ac12ae9f63c-webhook-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-nxt2j\" (UID: \"df7eb9de-4a8d-41bc-be97-0ac12ae9f63c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j"
Oct 31 00:22:39.872241 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.871334    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/df7eb9de-4a8d-41bc-be97-0ac12ae9f63c-metrics-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-nxt2j\" (UID: \"df7eb9de-4a8d-41bc-be97-0ac12ae9f63c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j"
Oct 31 00:22:39.875039 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.874992    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df7eb9de-4a8d-41bc-be97-0ac12ae9f63c-tmp\") pod \"opendatahub-operator-controller-manager-f465f7bb6-nxt2j\" (UID: \"df7eb9de-4a8d-41bc-be97-0ac12ae9f63c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j"
Oct 31 00:22:39.879235 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.879212    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j"]
Oct 31 00:22:39.901553 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.901522    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vbh5\" (UniqueName: \"kubernetes.io/projected/df7eb9de-4a8d-41bc-be97-0ac12ae9f63c-kube-api-access-5vbh5\") pod \"opendatahub-operator-controller-manager-f465f7bb6-nxt2j\" (UID: \"df7eb9de-4a8d-41bc-be97-0ac12ae9f63c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j"
Oct 31 00:22:39.969697 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:39.969598    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j"
Oct 31 00:22:40.330206 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:40.330120    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j"]
Oct 31 00:22:40.341533 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:22:40.341504    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf7eb9de_4a8d_41bc_be97_0ac12ae9f63c.slice/crio-f370dfcd05dfec2354456ff2927e7486f9402fc070d4c0980a85ba6f13c81cdc WatchSource:0}: Error finding container f370dfcd05dfec2354456ff2927e7486f9402fc070d4c0980a85ba6f13c81cdc: Status 404 returned error can't find the container with id f370dfcd05dfec2354456ff2927e7486f9402fc070d4c0980a85ba6f13c81cdc
Oct 31 00:22:40.752773 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:40.752737    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j" event={"ID":"df7eb9de-4a8d-41bc-be97-0ac12ae9f63c","Type":"ContainerStarted","Data":"f370dfcd05dfec2354456ff2927e7486f9402fc070d4c0980a85ba6f13c81cdc"}
Oct 31 00:22:43.764161 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:43.764127    4327 generic.go:356] "Generic (PLEG): container finished" podID="df7eb9de-4a8d-41bc-be97-0ac12ae9f63c" containerID="f7ce9e354d6f527e706f18501c807733c1acebfbda85556fbb9153747ee71bbf" exitCode=0
Oct 31 00:22:43.764502 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:43.764172    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j" event={"ID":"df7eb9de-4a8d-41bc-be97-0ac12ae9f63c","Type":"ContainerDied","Data":"f7ce9e354d6f527e706f18501c807733c1acebfbda85556fbb9153747ee71bbf"}
Oct 31 00:22:44.770346 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:44.770301    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j" event={"ID":"df7eb9de-4a8d-41bc-be97-0ac12ae9f63c","Type":"ContainerStarted","Data":"96470daa112798cfecdd12803f2b96c7b2da67855bcdc49452281d11102b2c42"}
Oct 31 00:22:44.770765 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:44.770405    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j"
Oct 31 00:22:44.873702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:44.873614    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j" podStartSLOduration=2.219647778 podStartE2EDuration="5.873597396s" totalImagesPullingTime="3.164812262s" totalInitContainerRuntime="489.137356ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:44.862119085 +0000 UTC m=+225.084154628" watchObservedRunningTime="2026-10-31 00:22:44.873597396 +0000 UTC m=+225.095632940"
Oct 31 00:22:45.290622 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:22:45.290584    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:22:55.776254 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:22:55.776214    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-nxt2j"
Oct 31 00:23:13.719860 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:23:13.719822    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-94bcdf98f-g9zcg"]
Oct 31 00:23:13.726432 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:23:13.726406    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-94bcdf98f-g9zcg"
Oct 31 00:23:13.743107 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:23:13.743078    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-94bcdf98f-g9zcg"]
Oct 31 00:23:13.817242 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:23:13.817205    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 31 00:23:13.817410 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:23:13.817246    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-dvh59\""
Oct 31 00:23:13.817410 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:23:13.817294    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 31 00:23:13.843046 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:23:13.843001    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmlj9\" (UniqueName: \"kubernetes.io/projected/3abd5af3-5337-4948-849b-b6d2976f269e-kube-api-access-cmlj9\") pod \"kube-auth-proxy-94bcdf98f-g9zcg\" (UID: \"3abd5af3-5337-4948-849b-b6d2976f269e\") " pod="openshift-ingress/kube-auth-proxy-94bcdf98f-g9zcg"
Oct 31 00:23:13.843221 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:23:13.843066    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3abd5af3-5337-4948-849b-b6d2976f269e-tls-certs\") pod \"kube-auth-proxy-94bcdf98f-g9zcg\" (UID: \"3abd5af3-5337-4948-849b-b6d2976f269e\") " pod="openshift-ingress/kube-auth-proxy-94bcdf98f-g9zcg"
Oct 31 00:23:13.843221 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:23:13.843158    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3abd5af3-5337-4948-849b-b6d2976f269e-tmp\") pod \"kube-auth-proxy-94bcdf98f-g9zcg\" (UID: \"3abd5af3-5337-4948-849b-b6d2976f269e\") " pod="openshift-ingress/kube-auth-proxy-94bcdf98f-g9zcg"
Oct 31 00:23:13.944217 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:23:13.944183    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmlj9\" (UniqueName: \"kubernetes.io/projected/3abd5af3-5337-4948-849b-b6d2976f269e-kube-api-access-cmlj9\") pod \"kube-auth-proxy-94bcdf98f-g9zcg\" (UID: \"3abd5af3-5337-4948-849b-b6d2976f269e\") " pod="openshift-ingress/kube-auth-proxy-94bcdf98f-g9zcg"
Oct 31 00:23:13.944391 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:23:13.944229    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3abd5af3-5337-4948-849b-b6d2976f269e-tls-certs\") pod \"kube-auth-proxy-94bcdf98f-g9zcg\" (UID: \"3abd5af3-5337-4948-849b-b6d2976f269e\") " pod="openshift-ingress/kube-auth-proxy-94bcdf98f-g9zcg"
Oct 31 00:23:13.944391 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:23:13.944260    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3abd5af3-5337-4948-849b-b6d2976f269e-tmp\") pod \"kube-auth-proxy-94bcdf98f-g9zcg\" (UID: \"3abd5af3-5337-4948-849b-b6d2976f269e\") " pod="openshift-ingress/kube-auth-proxy-94bcdf98f-g9zcg"
Oct 31 00:23:13.946720 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:23:13.946660    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3abd5af3-5337-4948-849b-b6d2976f269e-tmp\") pod \"kube-auth-proxy-94bcdf98f-g9zcg\" (UID: \"3abd5af3-5337-4948-849b-b6d2976f269e\") " pod="openshift-ingress/kube-auth-proxy-94bcdf98f-g9zcg"
Oct 31 00:23:13.946899 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:23:13.946879    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3abd5af3-5337-4948-849b-b6d2976f269e-tls-certs\") pod \"kube-auth-proxy-94bcdf98f-g9zcg\" (UID: \"3abd5af3-5337-4948-849b-b6d2976f269e\") " pod="openshift-ingress/kube-auth-proxy-94bcdf98f-g9zcg"
Oct 31 00:23:13.970515 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:23:13.970448    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmlj9\" (UniqueName: \"kubernetes.io/projected/3abd5af3-5337-4948-849b-b6d2976f269e-kube-api-access-cmlj9\") pod \"kube-auth-proxy-94bcdf98f-g9zcg\" (UID: \"3abd5af3-5337-4948-849b-b6d2976f269e\") " pod="openshift-ingress/kube-auth-proxy-94bcdf98f-g9zcg"
Oct 31 00:23:14.042814 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:23:14.042776    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-94bcdf98f-g9zcg"
Oct 31 00:23:14.339521 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:23:14.339498    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-94bcdf98f-g9zcg"]
Oct 31 00:23:14.342123 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:23:14.342096    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3abd5af3_5337_4948_849b_b6d2976f269e.slice/crio-00211a70d84985ca4e9ac4c4add1b55a9a5c1838d9c3aab727e6703fc0c37731 WatchSource:0}: Error finding container 00211a70d84985ca4e9ac4c4add1b55a9a5c1838d9c3aab727e6703fc0c37731: Status 404 returned error can't find the container with id 00211a70d84985ca4e9ac4c4add1b55a9a5c1838d9c3aab727e6703fc0c37731
Oct 31 00:23:14.875715 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:23:14.875659    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-94bcdf98f-g9zcg" event={"ID":"3abd5af3-5337-4948-849b-b6d2976f269e","Type":"ContainerStarted","Data":"00211a70d84985ca4e9ac4c4add1b55a9a5c1838d9c3aab727e6703fc0c37731"}
Oct 31 00:23:15.300035 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:23:15.299946    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:23:16.886437 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:23:16.886401    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-94bcdf98f-g9zcg" event={"ID":"3abd5af3-5337-4948-849b-b6d2976f269e","Type":"ContainerStarted","Data":"084477cf5e99913bd1c5091d7900a84316ea7f1b52dc63eee06d6f6c0622d590"}
Oct 31 00:23:16.931298 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:23:16.931251    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-94bcdf98f-g9zcg" podStartSLOduration=1.481812722 podStartE2EDuration="3.931236978s" totalImagesPullingTime="2.449424256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:16.930809937 +0000 UTC m=+257.152845484" watchObservedRunningTime="2026-10-31 00:23:16.931236978 +0000 UTC m=+257.153272517"
Oct 31 00:23:45.291203 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:23:45.291160    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:24:15.291649 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:24:15.291570    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:24:16.047255 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.047216    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"]
Oct 31 00:24:16.049374 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.049357    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"
Oct 31 00:24:16.112256 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.112221    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"]
Oct 31 00:24:16.151452 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.151421    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97aa8115-64f5-478f-93b1-276dc60acd6a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-frj8v\" (UID: \"97aa8115-64f5-478f-93b1-276dc60acd6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"
Oct 31 00:24:16.151633 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.151464    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97aa8115-64f5-478f-93b1-276dc60acd6a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-frj8v\" (UID: \"97aa8115-64f5-478f-93b1-276dc60acd6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"
Oct 31 00:24:16.151633 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.151491    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97aa8115-64f5-478f-93b1-276dc60acd6a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-frj8v\" (UID: \"97aa8115-64f5-478f-93b1-276dc60acd6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"
Oct 31 00:24:16.151633 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.151590    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97aa8115-64f5-478f-93b1-276dc60acd6a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-frj8v\" (UID: \"97aa8115-64f5-478f-93b1-276dc60acd6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"
Oct 31 00:24:16.151807 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.151641    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97aa8115-64f5-478f-93b1-276dc60acd6a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-frj8v\" (UID: \"97aa8115-64f5-478f-93b1-276dc60acd6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"
Oct 31 00:24:16.151807 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.151706    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97aa8115-64f5-478f-93b1-276dc60acd6a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-frj8v\" (UID: \"97aa8115-64f5-478f-93b1-276dc60acd6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"
Oct 31 00:24:16.151807 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.151730    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhk6x\" (UniqueName: \"kubernetes.io/projected/97aa8115-64f5-478f-93b1-276dc60acd6a-kube-api-access-zhk6x\") pod \"istiod-openshift-gateway-7ddd9cfdc8-frj8v\" (UID: \"97aa8115-64f5-478f-93b1-276dc60acd6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"
Oct 31 00:24:16.252989 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.252953    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97aa8115-64f5-478f-93b1-276dc60acd6a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-frj8v\" (UID: \"97aa8115-64f5-478f-93b1-276dc60acd6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"
Oct 31 00:24:16.252989 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.252991    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97aa8115-64f5-478f-93b1-276dc60acd6a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-frj8v\" (UID: \"97aa8115-64f5-478f-93b1-276dc60acd6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"
Oct 31 00:24:16.253204 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.253013    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97aa8115-64f5-478f-93b1-276dc60acd6a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-frj8v\" (UID: \"97aa8115-64f5-478f-93b1-276dc60acd6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"
Oct 31 00:24:16.253204 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.253129    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhk6x\" (UniqueName: \"kubernetes.io/projected/97aa8115-64f5-478f-93b1-276dc60acd6a-kube-api-access-zhk6x\") pod \"istiod-openshift-gateway-7ddd9cfdc8-frj8v\" (UID: \"97aa8115-64f5-478f-93b1-276dc60acd6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"
Oct 31 00:24:16.253307 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.253216    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97aa8115-64f5-478f-93b1-276dc60acd6a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-frj8v\" (UID: \"97aa8115-64f5-478f-93b1-276dc60acd6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"
Oct 31 00:24:16.253307 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.253272    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97aa8115-64f5-478f-93b1-276dc60acd6a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-frj8v\" (UID: \"97aa8115-64f5-478f-93b1-276dc60acd6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"
Oct 31 00:24:16.253398 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.253318    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97aa8115-64f5-478f-93b1-276dc60acd6a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-frj8v\" (UID: \"97aa8115-64f5-478f-93b1-276dc60acd6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"
Oct 31 00:24:16.254130 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.254103    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97aa8115-64f5-478f-93b1-276dc60acd6a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-frj8v\" (UID: \"97aa8115-64f5-478f-93b1-276dc60acd6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"
Oct 31 00:24:16.255415 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.255392    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97aa8115-64f5-478f-93b1-276dc60acd6a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-frj8v\" (UID: \"97aa8115-64f5-478f-93b1-276dc60acd6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"
Oct 31 00:24:16.255552 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.255473    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97aa8115-64f5-478f-93b1-276dc60acd6a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-frj8v\" (UID: \"97aa8115-64f5-478f-93b1-276dc60acd6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"
Oct 31 00:24:16.255875 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.255853    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97aa8115-64f5-478f-93b1-276dc60acd6a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-frj8v\" (UID: \"97aa8115-64f5-478f-93b1-276dc60acd6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"
Oct 31 00:24:16.255971 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.255854    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97aa8115-64f5-478f-93b1-276dc60acd6a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-frj8v\" (UID: \"97aa8115-64f5-478f-93b1-276dc60acd6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"
Oct 31 00:24:16.303226 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.303158    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97aa8115-64f5-478f-93b1-276dc60acd6a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-frj8v\" (UID: \"97aa8115-64f5-478f-93b1-276dc60acd6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"
Oct 31 00:24:16.303771 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.303748    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhk6x\" (UniqueName: \"kubernetes.io/projected/97aa8115-64f5-478f-93b1-276dc60acd6a-kube-api-access-zhk6x\") pod \"istiod-openshift-gateway-7ddd9cfdc8-frj8v\" (UID: \"97aa8115-64f5-478f-93b1-276dc60acd6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"
Oct 31 00:24:16.359617 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.359583    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"
Oct 31 00:24:16.686372 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.686335    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"]
Oct 31 00:24:16.695202 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:24:16.695172    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97aa8115_64f5_478f_93b1_276dc60acd6a.slice/crio-8483cd21f5c444e8398c130971cabfc75633180b2e07ca5c6fb8e2756cc55210 WatchSource:0}: Error finding container 8483cd21f5c444e8398c130971cabfc75633180b2e07ca5c6fb8e2756cc55210: Status 404 returned error can't find the container with id 8483cd21f5c444e8398c130971cabfc75633180b2e07ca5c6fb8e2756cc55210
Oct 31 00:24:16.697877 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.697835    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:24:16.697987 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:16.697947    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:24:17.114764 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:17.114725    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v" event={"ID":"97aa8115-64f5-478f-93b1-276dc60acd6a","Type":"ContainerStarted","Data":"674c0ac8015bf3f761a46fe09450a2f5c567b21c8fa8da1b33ff13293048c124"}
Oct 31 00:24:17.114977 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:17.114805    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v" event={"ID":"97aa8115-64f5-478f-93b1-276dc60acd6a","Type":"ContainerStarted","Data":"8483cd21f5c444e8398c130971cabfc75633180b2e07ca5c6fb8e2756cc55210"}
Oct 31 00:24:17.114977 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:17.114933    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"
Oct 31 00:24:17.116590 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:17.116563    4327 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-frj8v container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:24:17.116732 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:17.116610    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v" podUID="97aa8115-64f5-478f-93b1-276dc60acd6a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:24:17.200259 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:17.200204    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v" podStartSLOduration=1.20019069 podStartE2EDuration="1.20019069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:24:17.198171756 +0000 UTC m=+317.420207298" watchObservedRunningTime="2026-10-31 00:24:17.20019069 +0000 UTC m=+317.422226229"
Oct 31 00:24:18.119386 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:18.119359    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-frj8v"
Oct 31 00:24:18.334268 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:18.334239    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"]
Oct 31 00:24:18.334497 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:18.334460    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7" podUID="a25b686d-53ec-43f2-984b-bb6723f8fb50" containerName="discovery" containerID="cri-o://125f7f6fda7f9ec0fe76bd2a7c6d2800279dc3bb4641eb377d40d8eb6a5ae5df" gracePeriod=30
Oct 31 00:24:18.572304 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:18.572277    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:24:18.677403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:18.676086    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25b686d-53ec-43f2-984b-bb6723f8fb50-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25b686d-53ec-43f2-984b-bb6723f8fb50-local-certs\") pod \"a25b686d-53ec-43f2-984b-bb6723f8fb50\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") "
Oct 31 00:24:18.677403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:18.676174    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-token\" (UniqueName: \"kubernetes.io/projected/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-token\") pod \"a25b686d-53ec-43f2-984b-bb6723f8fb50\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") "
Oct 31 00:24:18.677403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:18.676245    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-csr-ca-configmap\") pod \"a25b686d-53ec-43f2-984b-bb6723f8fb50\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") "
Oct 31 00:24:18.677403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:18.676319    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a25b686d-53ec-43f2-984b-bb6723f8fb50-kube-api-access-hh5fv\" (UniqueName: \"kubernetes.io/projected/a25b686d-53ec-43f2-984b-bb6723f8fb50-kube-api-access-hh5fv\") pod \"a25b686d-53ec-43f2-984b-bb6723f8fb50\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") "
Oct 31 00:24:18.677403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:18.676365    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-kubeconfig\") pod \"a25b686d-53ec-43f2-984b-bb6723f8fb50\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") "
Oct 31 00:24:18.677403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:18.676407    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a25b686d-53ec-43f2-984b-bb6723f8fb50-cacerts\" (UniqueName: \"kubernetes.io/secret/a25b686d-53ec-43f2-984b-bb6723f8fb50-cacerts\") pod \"a25b686d-53ec-43f2-984b-bb6723f8fb50\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") "
Oct 31 00:24:18.677403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:18.676462    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-csr-dns-cert\") pod \"a25b686d-53ec-43f2-984b-bb6723f8fb50\" (UID: \"a25b686d-53ec-43f2-984b-bb6723f8fb50\") "
Oct 31 00:24:18.681324 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:18.681285    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-csr-ca-configmap" pod "a25b686d-53ec-43f2-984b-bb6723f8fb50" (UID: "a25b686d-53ec-43f2-984b-bb6723f8fb50"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:24:18.682326 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:18.682294    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-csr-dns-cert" pod "a25b686d-53ec-43f2-984b-bb6723f8fb50" (UID: "a25b686d-53ec-43f2-984b-bb6723f8fb50"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:24:18.682563 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:18.682532    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-kubeconfig" pod "a25b686d-53ec-43f2-984b-bb6723f8fb50" (UID: "a25b686d-53ec-43f2-984b-bb6723f8fb50"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:24:18.682706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:18.682592    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25b686d-53ec-43f2-984b-bb6723f8fb50-kube-api-access-hh5fv" pod "a25b686d-53ec-43f2-984b-bb6723f8fb50" (UID: "a25b686d-53ec-43f2-984b-bb6723f8fb50"). InnerVolumeSpecName "kube-api-access-hh5fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:24:18.682782 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:18.682698    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25b686d-53ec-43f2-984b-bb6723f8fb50-local-certs" pod "a25b686d-53ec-43f2-984b-bb6723f8fb50" (UID: "a25b686d-53ec-43f2-984b-bb6723f8fb50"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:18.683178 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:18.683158    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a25b686d-53ec-43f2-984b-bb6723f8fb50-cacerts" pod "a25b686d-53ec-43f2-984b-bb6723f8fb50" (UID: "a25b686d-53ec-43f2-984b-bb6723f8fb50"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:24:18.683378 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:18.683361    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-token" pod "a25b686d-53ec-43f2-984b-bb6723f8fb50" (UID: "a25b686d-53ec-43f2-984b-bb6723f8fb50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:24:18.777986 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:18.777953    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:24:18.777986 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:18.777982    4327 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-csr-ca-configmap\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:24:18.777986 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:18.777992    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh5fv\" (UniqueName: \"kubernetes.io/projected/a25b686d-53ec-43f2-984b-bb6723f8fb50-kube-api-access-hh5fv\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:24:18.778204 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:18.778002    4327 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-kubeconfig\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:24:18.778204 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:18.778011    4327 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a25b686d-53ec-43f2-984b-bb6723f8fb50-cacerts\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:24:18.778204 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:18.778019    4327 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a25b686d-53ec-43f2-984b-bb6723f8fb50-istio-csr-dns-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:24:18.778204 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:18.778028    4327 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25b686d-53ec-43f2-984b-bb6723f8fb50-local-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:24:19.123512 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:19.123475    4327 generic.go:356] "Generic (PLEG): container finished" podID="a25b686d-53ec-43f2-984b-bb6723f8fb50" containerID="125f7f6fda7f9ec0fe76bd2a7c6d2800279dc3bb4641eb377d40d8eb6a5ae5df" exitCode=0
Oct 31 00:24:19.123917 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:19.123532    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"
Oct 31 00:24:19.123917 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:19.123563    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7" event={"ID":"a25b686d-53ec-43f2-984b-bb6723f8fb50","Type":"ContainerDied","Data":"125f7f6fda7f9ec0fe76bd2a7c6d2800279dc3bb4641eb377d40d8eb6a5ae5df"}
Oct 31 00:24:19.123917 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:19.123622    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7" event={"ID":"a25b686d-53ec-43f2-984b-bb6723f8fb50","Type":"ContainerDied","Data":"a69aa733c43a90f5360961078e653bfdc20dd4ae3417cc72f00da79895f92d74"}
Oct 31 00:24:19.123917 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:19.123647    4327 scope.go:122] "RemoveContainer" containerID="125f7f6fda7f9ec0fe76bd2a7c6d2800279dc3bb4641eb377d40d8eb6a5ae5df"
Oct 31 00:24:19.134962 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:19.134945    4327 scope.go:122] "RemoveContainer" containerID="125f7f6fda7f9ec0fe76bd2a7c6d2800279dc3bb4641eb377d40d8eb6a5ae5df"
Oct 31 00:24:19.135720 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:24:19.135661    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"125f7f6fda7f9ec0fe76bd2a7c6d2800279dc3bb4641eb377d40d8eb6a5ae5df\": container with ID starting with 125f7f6fda7f9ec0fe76bd2a7c6d2800279dc3bb4641eb377d40d8eb6a5ae5df not found: ID does not exist" containerID="125f7f6fda7f9ec0fe76bd2a7c6d2800279dc3bb4641eb377d40d8eb6a5ae5df"
Oct 31 00:24:19.135812 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:19.135746    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"125f7f6fda7f9ec0fe76bd2a7c6d2800279dc3bb4641eb377d40d8eb6a5ae5df"} err="failed to get container status \"125f7f6fda7f9ec0fe76bd2a7c6d2800279dc3bb4641eb377d40d8eb6a5ae5df\": rpc error: code = NotFound desc = could not find container \"125f7f6fda7f9ec0fe76bd2a7c6d2800279dc3bb4641eb377d40d8eb6a5ae5df\": container with ID starting with 125f7f6fda7f9ec0fe76bd2a7c6d2800279dc3bb4641eb377d40d8eb6a5ae5df not found: ID does not exist"
Oct 31 00:24:19.178903 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:19.178868    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"]
Oct 31 00:24:19.190875 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:19.190841    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8dkw7"]
Oct 31 00:24:19.918263 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:24:19.918222    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a25b686d-53ec-43f2-984b-bb6723f8fb50" path="/var/lib/kubelet/pods/a25b686d-53ec-43f2-984b-bb6723f8fb50/volumes"
Oct 31 00:24:45.291001 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:24:45.290960    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:25:15.291771 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:25:15.291732    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:25:45.291053 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:25:45.290970    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:26:15.291505 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:26:15.291464    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:26:45.292435 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:26:45.292394    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:27:15.291517 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:27:15.291438    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:27:45.291756 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:27:45.291711    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:28:15.290561 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:28:15.290522    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:28:45.291112 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:28:45.291069    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:29:15.291462 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:29:15.291414    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:29:45.291090 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:29:45.291053    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:30:15.290912 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:30:15.290834    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:30:45.291037 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:30:45.290998    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:31:15.291327 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:31:15.291291    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:31:45.291152 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:31:45.291068    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:32:15.290920 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:32:15.290879    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:32:45.290978 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:32:45.290938    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:33:15.290750 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:33:15.290656    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:33:45.291824 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:33:45.291784    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:33:51.760454 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.760418    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pqsfr"]
Oct 31 00:33:51.760953 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.760717    4327 cpu_manager.go:414] "containerMap: removing container" podUID="a25b686d-53ec-43f2-984b-bb6723f8fb50" containerName="discovery"
Oct 31 00:33:51.760953 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.760730    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a25b686d-53ec-43f2-984b-bb6723f8fb50" containerName="discovery"
Oct 31 00:33:51.760953 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.760775    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="a25b686d-53ec-43f2-984b-bb6723f8fb50" containerName="discovery"
Oct 31 00:33:51.763538 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.763521    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-pqsfr"
Oct 31 00:33:51.778822 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.778800    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-d6q45\""
Oct 31 00:33:51.790961 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.790932    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 00:33:51.790961 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.790928    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 31 00:33:51.791200 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.790968    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 00:33:51.792534 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.792507    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pqsfr"]
Oct 31 00:33:51.797564 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.797537    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7khw\" (UniqueName: \"kubernetes.io/projected/681526a4-0e0c-4682-8653-dd0d67c74562-kube-api-access-q7khw\") pod \"limitador-limitador-74dfcfb894-pqsfr\" (UID: \"681526a4-0e0c-4682-8653-dd0d67c74562\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pqsfr"
Oct 31 00:33:51.797712 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.797592    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/681526a4-0e0c-4682-8653-dd0d67c74562-config-file\") pod \"limitador-limitador-74dfcfb894-pqsfr\" (UID: \"681526a4-0e0c-4682-8653-dd0d67c74562\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pqsfr"
Oct 31 00:33:51.828040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.828005    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-k7vdf"]
Oct 31 00:33:51.831408 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.831383    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k7vdf"
Oct 31 00:33:51.852304 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.852271    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-k7vdf"]
Oct 31 00:33:51.879023 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.878995    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 31 00:33:51.879023 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.878994    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 31 00:33:51.898557 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.898526    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0686e3b-1c7a-42f1-b7be-7c85553be299-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-k7vdf\" (UID: \"e0686e3b-1c7a-42f1-b7be-7c85553be299\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k7vdf"
Oct 31 00:33:51.898735 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.898573    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7khw\" (UniqueName: \"kubernetes.io/projected/681526a4-0e0c-4682-8653-dd0d67c74562-kube-api-access-q7khw\") pod \"limitador-limitador-74dfcfb894-pqsfr\" (UID: \"681526a4-0e0c-4682-8653-dd0d67c74562\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pqsfr"
Oct 31 00:33:51.898824 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.898758    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/681526a4-0e0c-4682-8653-dd0d67c74562-config-file\") pod \"limitador-limitador-74dfcfb894-pqsfr\" (UID: \"681526a4-0e0c-4682-8653-dd0d67c74562\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pqsfr"
Oct 31 00:33:51.898824 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.898814    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5rkr\" (UniqueName: \"kubernetes.io/projected/e0686e3b-1c7a-42f1-b7be-7c85553be299-kube-api-access-n5rkr\") pod \"kuadrant-console-plugin-66c4fb575d-k7vdf\" (UID: \"e0686e3b-1c7a-42f1-b7be-7c85553be299\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k7vdf"
Oct 31 00:33:51.898924 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.898839    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e0686e3b-1c7a-42f1-b7be-7c85553be299-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-k7vdf\" (UID: \"e0686e3b-1c7a-42f1-b7be-7c85553be299\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k7vdf"
Oct 31 00:33:51.899387 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.899369    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/681526a4-0e0c-4682-8653-dd0d67c74562-config-file\") pod \"limitador-limitador-74dfcfb894-pqsfr\" (UID: \"681526a4-0e0c-4682-8653-dd0d67c74562\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pqsfr"
Oct 31 00:33:51.906798 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.906767    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pqsfr"]
Oct 31 00:33:51.922734 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.922669    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7khw\" (UniqueName: \"kubernetes.io/projected/681526a4-0e0c-4682-8653-dd0d67c74562-kube-api-access-q7khw\") pod \"limitador-limitador-74dfcfb894-pqsfr\" (UID: \"681526a4-0e0c-4682-8653-dd0d67c74562\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pqsfr"
Oct 31 00:33:51.999621 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.999577    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5rkr\" (UniqueName: \"kubernetes.io/projected/e0686e3b-1c7a-42f1-b7be-7c85553be299-kube-api-access-n5rkr\") pod \"kuadrant-console-plugin-66c4fb575d-k7vdf\" (UID: \"e0686e3b-1c7a-42f1-b7be-7c85553be299\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k7vdf"
Oct 31 00:33:51.999891 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.999752    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e0686e3b-1c7a-42f1-b7be-7c85553be299-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-k7vdf\" (UID: \"e0686e3b-1c7a-42f1-b7be-7c85553be299\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k7vdf"
Oct 31 00:33:51.999891 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:51.999798    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0686e3b-1c7a-42f1-b7be-7c85553be299-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-k7vdf\" (UID: \"e0686e3b-1c7a-42f1-b7be-7c85553be299\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k7vdf"
Oct 31 00:33:52.000526 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:52.000502    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e0686e3b-1c7a-42f1-b7be-7c85553be299-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-k7vdf\" (UID: \"e0686e3b-1c7a-42f1-b7be-7c85553be299\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k7vdf"
Oct 31 00:33:52.002198 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:52.002176    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0686e3b-1c7a-42f1-b7be-7c85553be299-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-k7vdf\" (UID: \"e0686e3b-1c7a-42f1-b7be-7c85553be299\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k7vdf"
Oct 31 00:33:52.023656 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:52.023588    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5rkr\" (UniqueName: \"kubernetes.io/projected/e0686e3b-1c7a-42f1-b7be-7c85553be299-kube-api-access-n5rkr\") pod \"kuadrant-console-plugin-66c4fb575d-k7vdf\" (UID: \"e0686e3b-1c7a-42f1-b7be-7c85553be299\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k7vdf"
Oct 31 00:33:52.078052 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:52.078011    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-pqsfr"
Oct 31 00:33:52.161068 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:52.161033    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k7vdf"
Oct 31 00:33:52.372282 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:52.372259    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pqsfr"]
Oct 31 00:33:52.373612 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:33:52.373583    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681526a4_0e0c_4682_8653_dd0d67c74562.slice/crio-cae1931708b63edd98f8e637e93e34f904c57e95527a261e0c21683285646e48 WatchSource:0}: Error finding container cae1931708b63edd98f8e637e93e34f904c57e95527a261e0c21683285646e48: Status 404 returned error can't find the container with id cae1931708b63edd98f8e637e93e34f904c57e95527a261e0c21683285646e48
Oct 31 00:33:52.375390 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:52.375372    4327 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:33:52.456701 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:33:52.456654    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0686e3b_1c7a_42f1_b7be_7c85553be299.slice/crio-edb14e7bea0966a4f427420daf026627d05d30ced9c9a6a213bed93b95e8434a WatchSource:0}: Error finding container edb14e7bea0966a4f427420daf026627d05d30ced9c9a6a213bed93b95e8434a: Status 404 returned error can't find the container with id edb14e7bea0966a4f427420daf026627d05d30ced9c9a6a213bed93b95e8434a
Oct 31 00:33:52.457102 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:52.457079    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-k7vdf"]
Oct 31 00:33:53.318368 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:53.318328    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-pqsfr" event={"ID":"681526a4-0e0c-4682-8653-dd0d67c74562","Type":"ContainerStarted","Data":"cae1931708b63edd98f8e637e93e34f904c57e95527a261e0c21683285646e48"}
Oct 31 00:33:53.319868 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:53.319794    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k7vdf" event={"ID":"e0686e3b-1c7a-42f1-b7be-7c85553be299","Type":"ContainerStarted","Data":"edb14e7bea0966a4f427420daf026627d05d30ced9c9a6a213bed93b95e8434a"}
Oct 31 00:33:53.403278 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:53.401830    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nrpnv"]
Oct 31 00:33:53.406240 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:53.406050    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nrpnv"
Oct 31 00:33:53.411136 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:53.411109    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s9xt\" (UniqueName: \"kubernetes.io/projected/9edd5747-b0b2-4d57-a74c-82e5a425a8ee-kube-api-access-7s9xt\") pod \"authorino-56c6c9ddd5-nrpnv\" (UID: \"9edd5747-b0b2-4d57-a74c-82e5a425a8ee\") " pod="kuadrant-system/authorino-56c6c9ddd5-nrpnv"
Oct 31 00:33:53.425782 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:53.425751    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nrpnv"]
Oct 31 00:33:53.436673 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:53.436643    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-t5h5b\""
Oct 31 00:33:53.512002 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:53.511615    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s9xt\" (UniqueName: \"kubernetes.io/projected/9edd5747-b0b2-4d57-a74c-82e5a425a8ee-kube-api-access-7s9xt\") pod \"authorino-56c6c9ddd5-nrpnv\" (UID: \"9edd5747-b0b2-4d57-a74c-82e5a425a8ee\") " pod="kuadrant-system/authorino-56c6c9ddd5-nrpnv"
Oct 31 00:33:53.538978 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:53.538905    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s9xt\" (UniqueName: \"kubernetes.io/projected/9edd5747-b0b2-4d57-a74c-82e5a425a8ee-kube-api-access-7s9xt\") pod \"authorino-56c6c9ddd5-nrpnv\" (UID: \"9edd5747-b0b2-4d57-a74c-82e5a425a8ee\") " pod="kuadrant-system/authorino-56c6c9ddd5-nrpnv"
Oct 31 00:33:53.724202 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:53.724162    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nrpnv"
Oct 31 00:33:54.023330 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:54.023300    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nrpnv"]
Oct 31 00:33:54.327791 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:54.327714    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nrpnv" event={"ID":"9edd5747-b0b2-4d57-a74c-82e5a425a8ee","Type":"ContainerStarted","Data":"a00673b00ede3d7923719ea7320aeab4b223d670f56080ffdb363e097ddcd2cf"}
Oct 31 00:33:55.337274 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:55.337236    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-pqsfr" event={"ID":"681526a4-0e0c-4682-8653-dd0d67c74562","Type":"ContainerStarted","Data":"20d58bfc7e9d94d2aa051871229038ebd3db0356c4d6ecbf68b86d51a2203a94"}
Oct 31 00:33:55.337784 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:55.337368    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-pqsfr"
Oct 31 00:33:55.380669 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:55.380598    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-pqsfr" podStartSLOduration=1.551375582 podStartE2EDuration="4.380578597s" totalImagesPullingTime="2.829203015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:55.379153329 +0000 UTC m=+895.601188870" watchObservedRunningTime="2026-10-31 00:33:55.380578597 +0000 UTC m=+895.602614140"
Oct 31 00:33:56.344993 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:56.344898    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nrpnv" event={"ID":"9edd5747-b0b2-4d57-a74c-82e5a425a8ee","Type":"ContainerStarted","Data":"1f0f9db77631b4f96f96263b8240fc213f96e7057dd880dd6c4f03d09ac294f9"}
Oct 31 00:33:56.385324 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:56.385266    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-nrpnv" podStartSLOduration=1.370977288 podStartE2EDuration="3.3852497s" totalImagesPullingTime="2.014272412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:56.384631538 +0000 UTC m=+896.606667081" watchObservedRunningTime="2026-10-31 00:33:56.3852497 +0000 UTC m=+896.607285220"
Oct 31 00:33:58.355138 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:58.355104    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k7vdf" event={"ID":"e0686e3b-1c7a-42f1-b7be-7c85553be299","Type":"ContainerStarted","Data":"e0fd66cec75825a8e05961eae4a6c18aba7d0cb0762f170e4aae05a945b4c1e4"}
Oct 31 00:33:58.396446 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:33:58.396389    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k7vdf" podStartSLOduration=1.902537116 podStartE2EDuration="7.396372435s" totalImagesPullingTime="5.493835319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:58.395644378 +0000 UTC m=+898.617679919" watchObservedRunningTime="2026-10-31 00:33:58.396372435 +0000 UTC m=+898.618407976"
Oct 31 00:34:06.346773 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:06.346743    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-pqsfr"
Oct 31 00:34:07.162132 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:07.162101    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pqsfr"]
Oct 31 00:34:07.162319 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:07.162288    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-pqsfr" podUID="681526a4-0e0c-4682-8653-dd0d67c74562" containerName="limitador" containerID="cri-o://20d58bfc7e9d94d2aa051871229038ebd3db0356c4d6ecbf68b86d51a2203a94" gracePeriod=30
Oct 31 00:34:07.710940 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:07.710914    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-pqsfr"
Oct 31 00:34:07.840743 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:07.840630    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/681526a4-0e0c-4682-8653-dd0d67c74562-config-file\" (UniqueName: \"kubernetes.io/configmap/681526a4-0e0c-4682-8653-dd0d67c74562-config-file\") pod \"681526a4-0e0c-4682-8653-dd0d67c74562\" (UID: \"681526a4-0e0c-4682-8653-dd0d67c74562\") "
Oct 31 00:34:07.840743 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:07.840732    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/681526a4-0e0c-4682-8653-dd0d67c74562-kube-api-access-q7khw\" (UniqueName: \"kubernetes.io/projected/681526a4-0e0c-4682-8653-dd0d67c74562-kube-api-access-q7khw\") pod \"681526a4-0e0c-4682-8653-dd0d67c74562\" (UID: \"681526a4-0e0c-4682-8653-dd0d67c74562\") "
Oct 31 00:34:07.841053 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:07.841026    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/681526a4-0e0c-4682-8653-dd0d67c74562-config-file" pod "681526a4-0e0c-4682-8653-dd0d67c74562" (UID: "681526a4-0e0c-4682-8653-dd0d67c74562"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:34:07.842872 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:07.842845    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681526a4-0e0c-4682-8653-dd0d67c74562-kube-api-access-q7khw" pod "681526a4-0e0c-4682-8653-dd0d67c74562" (UID: "681526a4-0e0c-4682-8653-dd0d67c74562"). InnerVolumeSpecName "kube-api-access-q7khw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:34:07.941640 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:07.941608    4327 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/681526a4-0e0c-4682-8653-dd0d67c74562-config-file\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:34:07.941640 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:07.941634    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7khw\" (UniqueName: \"kubernetes.io/projected/681526a4-0e0c-4682-8653-dd0d67c74562-kube-api-access-q7khw\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:34:08.394973 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:08.394938    4327 generic.go:356] "Generic (PLEG): container finished" podID="681526a4-0e0c-4682-8653-dd0d67c74562" containerID="20d58bfc7e9d94d2aa051871229038ebd3db0356c4d6ecbf68b86d51a2203a94" exitCode=0
Oct 31 00:34:08.395144 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:08.394981    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-pqsfr" event={"ID":"681526a4-0e0c-4682-8653-dd0d67c74562","Type":"ContainerDied","Data":"20d58bfc7e9d94d2aa051871229038ebd3db0356c4d6ecbf68b86d51a2203a94"}
Oct 31 00:34:08.395144 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:08.394986    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-pqsfr"
Oct 31 00:34:08.395144 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:08.395021    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-pqsfr" event={"ID":"681526a4-0e0c-4682-8653-dd0d67c74562","Type":"ContainerDied","Data":"cae1931708b63edd98f8e637e93e34f904c57e95527a261e0c21683285646e48"}
Oct 31 00:34:08.395144 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:08.395042    4327 scope.go:122] "RemoveContainer" containerID="20d58bfc7e9d94d2aa051871229038ebd3db0356c4d6ecbf68b86d51a2203a94"
Oct 31 00:34:08.402948 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:08.402929    4327 scope.go:122] "RemoveContainer" containerID="20d58bfc7e9d94d2aa051871229038ebd3db0356c4d6ecbf68b86d51a2203a94"
Oct 31 00:34:08.403243 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:34:08.403224    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d58bfc7e9d94d2aa051871229038ebd3db0356c4d6ecbf68b86d51a2203a94\": container with ID starting with 20d58bfc7e9d94d2aa051871229038ebd3db0356c4d6ecbf68b86d51a2203a94 not found: ID does not exist" containerID="20d58bfc7e9d94d2aa051871229038ebd3db0356c4d6ecbf68b86d51a2203a94"
Oct 31 00:34:08.403291 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:08.403252    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d58bfc7e9d94d2aa051871229038ebd3db0356c4d6ecbf68b86d51a2203a94"} err="failed to get container status \"20d58bfc7e9d94d2aa051871229038ebd3db0356c4d6ecbf68b86d51a2203a94\": rpc error: code = NotFound desc = could not find container \"20d58bfc7e9d94d2aa051871229038ebd3db0356c4d6ecbf68b86d51a2203a94\": container with ID starting with 20d58bfc7e9d94d2aa051871229038ebd3db0356c4d6ecbf68b86d51a2203a94 not found: ID does not exist"
Oct 31 00:34:08.446331 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:08.446302    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pqsfr"]
Oct 31 00:34:08.458751 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:08.458725    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pqsfr"]
Oct 31 00:34:09.600977 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:09.600943    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-24d65"]
Oct 31 00:34:09.601362 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:09.601235    4327 cpu_manager.go:414] "containerMap: removing container" podUID="681526a4-0e0c-4682-8653-dd0d67c74562" containerName="limitador"
Oct 31 00:34:09.601362 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:09.601247    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681526a4-0e0c-4682-8653-dd0d67c74562" containerName="limitador"
Oct 31 00:34:09.601362 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:09.601314    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="681526a4-0e0c-4682-8653-dd0d67c74562" containerName="limitador"
Oct 31 00:34:09.605610 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:09.605592    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-24d65"
Oct 31 00:34:09.626959 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:09.626927    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 31 00:34:09.627346 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:09.627329    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:34:09.627479 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:09.627381    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-sth9p\""
Oct 31 00:34:09.627634 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:09.627534    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:34:09.628723 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:09.628520    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-24d65"]
Oct 31 00:34:09.657792 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:09.657761    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96pwg\" (UniqueName: \"kubernetes.io/projected/c50004be-2445-40ac-a248-ef5d679f9792-kube-api-access-96pwg\") pod \"postgres-7cdbf6dc44-24d65\" (UID: \"c50004be-2445-40ac-a248-ef5d679f9792\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-24d65"
Oct 31 00:34:09.657951 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:09.657829    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c50004be-2445-40ac-a248-ef5d679f9792-data\") pod \"postgres-7cdbf6dc44-24d65\" (UID: \"c50004be-2445-40ac-a248-ef5d679f9792\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-24d65"
Oct 31 00:34:09.759104 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:09.759063    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96pwg\" (UniqueName: \"kubernetes.io/projected/c50004be-2445-40ac-a248-ef5d679f9792-kube-api-access-96pwg\") pod \"postgres-7cdbf6dc44-24d65\" (UID: \"c50004be-2445-40ac-a248-ef5d679f9792\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-24d65"
Oct 31 00:34:09.759274 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:09.759122    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c50004be-2445-40ac-a248-ef5d679f9792-data\") pod \"postgres-7cdbf6dc44-24d65\" (UID: \"c50004be-2445-40ac-a248-ef5d679f9792\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-24d65"
Oct 31 00:34:09.759445 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:09.759431    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c50004be-2445-40ac-a248-ef5d679f9792-data\") pod \"postgres-7cdbf6dc44-24d65\" (UID: \"c50004be-2445-40ac-a248-ef5d679f9792\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-24d65"
Oct 31 00:34:09.783007 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:09.782982    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96pwg\" (UniqueName: \"kubernetes.io/projected/c50004be-2445-40ac-a248-ef5d679f9792-kube-api-access-96pwg\") pod \"postgres-7cdbf6dc44-24d65\" (UID: \"c50004be-2445-40ac-a248-ef5d679f9792\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-24d65"
Oct 31 00:34:09.918272 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:09.918194    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="681526a4-0e0c-4682-8653-dd0d67c74562" path="/var/lib/kubelet/pods/681526a4-0e0c-4682-8653-dd0d67c74562/volumes"
Oct 31 00:34:09.919483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:09.919465    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-24d65"
Oct 31 00:34:10.212519 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:10.212491    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-24d65"]
Oct 31 00:34:10.213855 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:34:10.213828    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc50004be_2445_40ac_a248_ef5d679f9792.slice/crio-797b8f5cce2c6dcddb8ca81ec7465d04cb546109b1913b351d750ba97005797f WatchSource:0}: Error finding container 797b8f5cce2c6dcddb8ca81ec7465d04cb546109b1913b351d750ba97005797f: Status 404 returned error can't find the container with id 797b8f5cce2c6dcddb8ca81ec7465d04cb546109b1913b351d750ba97005797f
Oct 31 00:34:10.404752 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:10.404713    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-24d65" event={"ID":"c50004be-2445-40ac-a248-ef5d679f9792","Type":"ContainerStarted","Data":"797b8f5cce2c6dcddb8ca81ec7465d04cb546109b1913b351d750ba97005797f"}
Oct 31 00:34:15.291623 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:34:15.291492    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:34:15.428501 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:15.428467    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-24d65" event={"ID":"c50004be-2445-40ac-a248-ef5d679f9792","Type":"ContainerStarted","Data":"5af4efe4b20802536a2b22cd4ae18a3f4452325def767a5431ca0c68c63dfb2e"}
Oct 31 00:34:15.428707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:15.428557    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-24d65"
Oct 31 00:34:15.469091 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:15.469041    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-24d65" podStartSLOduration=1.80071965 podStartE2EDuration="6.469028818s" totalImagesPullingTime="4.668309168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:15.46836351 +0000 UTC m=+915.690399051" watchObservedRunningTime="2026-10-31 00:34:15.469028818 +0000 UTC m=+915.691064360"
Oct 31 00:34:21.461729 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:21.461700    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-24d65"
Oct 31 00:34:28.681612 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:28.681570    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-2fksl"]
Oct 31 00:34:28.975879 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:28.975790    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-2fksl"]
Oct 31 00:34:28.976103 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:28.975976    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-2fksl"
Oct 31 00:34:29.003347 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.003311    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 31 00:34:29.003347 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.003345    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 00:34:29.003542 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.003347    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 00:34:29.003542 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.003348    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 31 00:34:29.003542 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.003385    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 31 00:34:29.003658 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.003554    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jtxr8\""
Oct 31 00:34:29.044795 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.044758    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9c8d8b4d5-frhlg"]
Oct 31 00:34:29.056129 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.056102    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9c8d8b4d5-frhlg"
Oct 31 00:34:29.070927 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.070891    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9c8d8b4d5-frhlg"]
Oct 31 00:34:29.125328 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.125288    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dcc5c11c-f607-4a2e-bdaf-988dd9416309-webhook-cert\") pod \"maas-controller-579cf7c6d-2fksl\" (UID: \"dcc5c11c-f607-4a2e-bdaf-988dd9416309\") " pod="opendatahub/maas-controller-579cf7c6d-2fksl"
Oct 31 00:34:29.125503 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.125451    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb82b\" (UniqueName: \"kubernetes.io/projected/dcc5c11c-f607-4a2e-bdaf-988dd9416309-kube-api-access-kb82b\") pod \"maas-controller-579cf7c6d-2fksl\" (UID: \"dcc5c11c-f607-4a2e-bdaf-988dd9416309\") " pod="opendatahub/maas-controller-579cf7c6d-2fksl"
Oct 31 00:34:29.125567 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.125506    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcc5c11c-f607-4a2e-bdaf-988dd9416309-metrics-certs\") pod \"maas-controller-579cf7c6d-2fksl\" (UID: \"dcc5c11c-f607-4a2e-bdaf-988dd9416309\") " pod="opendatahub/maas-controller-579cf7c6d-2fksl"
Oct 31 00:34:29.226292 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.226214    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql49d\" (UniqueName: \"kubernetes.io/projected/b5773d6f-3b5a-4e78-9d77-858c15e04639-kube-api-access-ql49d\") pod \"maas-controller-9c8d8b4d5-frhlg\" (UID: \"b5773d6f-3b5a-4e78-9d77-858c15e04639\") " pod="opendatahub/maas-controller-9c8d8b4d5-frhlg"
Oct 31 00:34:29.226292 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.226251    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb82b\" (UniqueName: \"kubernetes.io/projected/dcc5c11c-f607-4a2e-bdaf-988dd9416309-kube-api-access-kb82b\") pod \"maas-controller-579cf7c6d-2fksl\" (UID: \"dcc5c11c-f607-4a2e-bdaf-988dd9416309\") " pod="opendatahub/maas-controller-579cf7c6d-2fksl"
Oct 31 00:34:29.226292 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.226274    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcc5c11c-f607-4a2e-bdaf-988dd9416309-metrics-certs\") pod \"maas-controller-579cf7c6d-2fksl\" (UID: \"dcc5c11c-f607-4a2e-bdaf-988dd9416309\") " pod="opendatahub/maas-controller-579cf7c6d-2fksl"
Oct 31 00:34:29.226582 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.226332    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5773d6f-3b5a-4e78-9d77-858c15e04639-webhook-cert\") pod \"maas-controller-9c8d8b4d5-frhlg\" (UID: \"b5773d6f-3b5a-4e78-9d77-858c15e04639\") " pod="opendatahub/maas-controller-9c8d8b4d5-frhlg"
Oct 31 00:34:29.226582 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.226393    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dcc5c11c-f607-4a2e-bdaf-988dd9416309-webhook-cert\") pod \"maas-controller-579cf7c6d-2fksl\" (UID: \"dcc5c11c-f607-4a2e-bdaf-988dd9416309\") " pod="opendatahub/maas-controller-579cf7c6d-2fksl"
Oct 31 00:34:29.226582 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.226450    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5773d6f-3b5a-4e78-9d77-858c15e04639-metrics-certs\") pod \"maas-controller-9c8d8b4d5-frhlg\" (UID: \"b5773d6f-3b5a-4e78-9d77-858c15e04639\") " pod="opendatahub/maas-controller-9c8d8b4d5-frhlg"
Oct 31 00:34:29.228921 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.228899    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dcc5c11c-f607-4a2e-bdaf-988dd9416309-webhook-cert\") pod \"maas-controller-579cf7c6d-2fksl\" (UID: \"dcc5c11c-f607-4a2e-bdaf-988dd9416309\") " pod="opendatahub/maas-controller-579cf7c6d-2fksl"
Oct 31 00:34:29.229013 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.228897    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcc5c11c-f607-4a2e-bdaf-988dd9416309-metrics-certs\") pod \"maas-controller-579cf7c6d-2fksl\" (UID: \"dcc5c11c-f607-4a2e-bdaf-988dd9416309\") " pod="opendatahub/maas-controller-579cf7c6d-2fksl"
Oct 31 00:34:29.250451 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.250421    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb82b\" (UniqueName: \"kubernetes.io/projected/dcc5c11c-f607-4a2e-bdaf-988dd9416309-kube-api-access-kb82b\") pod \"maas-controller-579cf7c6d-2fksl\" (UID: \"dcc5c11c-f607-4a2e-bdaf-988dd9416309\") " pod="opendatahub/maas-controller-579cf7c6d-2fksl"
Oct 31 00:34:29.310844 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.310806    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-2fksl"
Oct 31 00:34:29.327115 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.327087    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5773d6f-3b5a-4e78-9d77-858c15e04639-metrics-certs\") pod \"maas-controller-9c8d8b4d5-frhlg\" (UID: \"b5773d6f-3b5a-4e78-9d77-858c15e04639\") " pod="opendatahub/maas-controller-9c8d8b4d5-frhlg"
Oct 31 00:34:29.327236 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.327134    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql49d\" (UniqueName: \"kubernetes.io/projected/b5773d6f-3b5a-4e78-9d77-858c15e04639-kube-api-access-ql49d\") pod \"maas-controller-9c8d8b4d5-frhlg\" (UID: \"b5773d6f-3b5a-4e78-9d77-858c15e04639\") " pod="opendatahub/maas-controller-9c8d8b4d5-frhlg"
Oct 31 00:34:29.327236 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.327156    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5773d6f-3b5a-4e78-9d77-858c15e04639-webhook-cert\") pod \"maas-controller-9c8d8b4d5-frhlg\" (UID: \"b5773d6f-3b5a-4e78-9d77-858c15e04639\") " pod="opendatahub/maas-controller-9c8d8b4d5-frhlg"
Oct 31 00:34:29.329544 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.329518    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5773d6f-3b5a-4e78-9d77-858c15e04639-metrics-certs\") pod \"maas-controller-9c8d8b4d5-frhlg\" (UID: \"b5773d6f-3b5a-4e78-9d77-858c15e04639\") " pod="opendatahub/maas-controller-9c8d8b4d5-frhlg"
Oct 31 00:34:29.330020 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.329711    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5773d6f-3b5a-4e78-9d77-858c15e04639-webhook-cert\") pod \"maas-controller-9c8d8b4d5-frhlg\" (UID: \"b5773d6f-3b5a-4e78-9d77-858c15e04639\") " pod="opendatahub/maas-controller-9c8d8b4d5-frhlg"
Oct 31 00:34:29.351191 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.351167    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql49d\" (UniqueName: \"kubernetes.io/projected/b5773d6f-3b5a-4e78-9d77-858c15e04639-kube-api-access-ql49d\") pod \"maas-controller-9c8d8b4d5-frhlg\" (UID: \"b5773d6f-3b5a-4e78-9d77-858c15e04639\") " pod="opendatahub/maas-controller-9c8d8b4d5-frhlg"
Oct 31 00:34:29.379022 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.378990    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9c8d8b4d5-frhlg"
Oct 31 00:34:29.620431 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.620405    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-2fksl"]
Oct 31 00:34:29.621826 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:34:29.621796    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcc5c11c_f607_4a2e_bdaf_988dd9416309.slice/crio-b4c6f1452bd0502ba6cb5a3e73ef1eafc8a6ddfdc775088e569285649a20c7bb WatchSource:0}: Error finding container b4c6f1452bd0502ba6cb5a3e73ef1eafc8a6ddfdc775088e569285649a20c7bb: Status 404 returned error can't find the container with id b4c6f1452bd0502ba6cb5a3e73ef1eafc8a6ddfdc775088e569285649a20c7bb
Oct 31 00:34:29.695208 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:29.695184    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9c8d8b4d5-frhlg"]
Oct 31 00:34:29.696116 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:34:29.696094    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5773d6f_3b5a_4e78_9d77_858c15e04639.slice/crio-3320ad6ebde556cbdd6de83054cf3388b9ee459d51ea769182f51a321dae96ef WatchSource:0}: Error finding container 3320ad6ebde556cbdd6de83054cf3388b9ee459d51ea769182f51a321dae96ef: Status 404 returned error can't find the container with id 3320ad6ebde556cbdd6de83054cf3388b9ee459d51ea769182f51a321dae96ef
Oct 31 00:34:30.492151 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:30.492115    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-2fksl" event={"ID":"dcc5c11c-f607-4a2e-bdaf-988dd9416309","Type":"ContainerStarted","Data":"b4c6f1452bd0502ba6cb5a3e73ef1eafc8a6ddfdc775088e569285649a20c7bb"}
Oct 31 00:34:30.494183 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:30.494097    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9c8d8b4d5-frhlg" event={"ID":"b5773d6f-3b5a-4e78-9d77-858c15e04639","Type":"ContainerStarted","Data":"3320ad6ebde556cbdd6de83054cf3388b9ee459d51ea769182f51a321dae96ef"}
Oct 31 00:34:32.505215 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:32.505177    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9c8d8b4d5-frhlg" event={"ID":"b5773d6f-3b5a-4e78-9d77-858c15e04639","Type":"ContainerStarted","Data":"f3a6e1abd5a80aec35338046a0348d14d2f3b5322577c0196a7f18fd93cdf47c"}
Oct 31 00:34:32.505662 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:32.505248    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9c8d8b4d5-frhlg"
Oct 31 00:34:32.506909 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:32.506878    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-2fksl" event={"ID":"dcc5c11c-f607-4a2e-bdaf-988dd9416309","Type":"ContainerStarted","Data":"70b9fefd9301243962826061b1ad761999642826c7e492e777834bacb18e158e"}
Oct 31 00:34:32.507031 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:32.507004    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-2fksl"
Oct 31 00:34:32.564035 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:32.563944    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-9c8d8b4d5-frhlg" podStartSLOduration=0.908769658 podStartE2EDuration="3.563931193s" totalImagesPullingTime="2.655161535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:32.561963912 +0000 UTC m=+932.783999466" watchObservedRunningTime="2026-10-31 00:34:32.563931193 +0000 UTC m=+932.785966729"
Oct 31 00:34:32.607175 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:32.607118    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-2fksl" podStartSLOduration=1.924703214 podStartE2EDuration="4.607102747s" totalImagesPullingTime="2.682399533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:32.606081182 +0000 UTC m=+932.828116727" watchObservedRunningTime="2026-10-31 00:34:32.607102747 +0000 UTC m=+932.829138289"
Oct 31 00:34:43.518140 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:43.518064    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-2fksl"
Oct 31 00:34:43.518140 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:43.518139    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9c8d8b4d5-frhlg"
Oct 31 00:34:43.634356 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:43.634323    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-2fksl"]
Oct 31 00:34:43.634569 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:43.634542    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-2fksl" podUID="dcc5c11c-f607-4a2e-bdaf-988dd9416309" containerName="manager" containerID="cri-o://70b9fefd9301243962826061b1ad761999642826c7e492e777834bacb18e158e" gracePeriod=30
Oct 31 00:34:43.870052 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:43.870030    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-2fksl"
Oct 31 00:34:43.948906 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:43.948868    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcc5c11c-f607-4a2e-bdaf-988dd9416309-kube-api-access-kb82b\" (UniqueName: \"kubernetes.io/projected/dcc5c11c-f607-4a2e-bdaf-988dd9416309-kube-api-access-kb82b\") pod \"dcc5c11c-f607-4a2e-bdaf-988dd9416309\" (UID: \"dcc5c11c-f607-4a2e-bdaf-988dd9416309\") "
Oct 31 00:34:43.948906 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:43.948905    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcc5c11c-f607-4a2e-bdaf-988dd9416309-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcc5c11c-f607-4a2e-bdaf-988dd9416309-metrics-certs\") pod \"dcc5c11c-f607-4a2e-bdaf-988dd9416309\" (UID: \"dcc5c11c-f607-4a2e-bdaf-988dd9416309\") "
Oct 31 00:34:43.951211 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:43.951175    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc5c11c-f607-4a2e-bdaf-988dd9416309-kube-api-access-kb82b" pod "dcc5c11c-f607-4a2e-bdaf-988dd9416309" (UID: "dcc5c11c-f607-4a2e-bdaf-988dd9416309"). InnerVolumeSpecName "kube-api-access-kb82b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:34:43.951211 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:43.951194    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcc5c11c-f607-4a2e-bdaf-988dd9416309-metrics-certs" pod "dcc5c11c-f607-4a2e-bdaf-988dd9416309" (UID: "dcc5c11c-f607-4a2e-bdaf-988dd9416309"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:34:44.049422 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:44.049385    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcc5c11c-f607-4a2e-bdaf-988dd9416309-webhook-cert\" (UniqueName: \"kubernetes.io/secret/dcc5c11c-f607-4a2e-bdaf-988dd9416309-webhook-cert\") pod \"dcc5c11c-f607-4a2e-bdaf-988dd9416309\" (UID: \"dcc5c11c-f607-4a2e-bdaf-988dd9416309\") "
Oct 31 00:34:44.049596 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:44.049572    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb82b\" (UniqueName: \"kubernetes.io/projected/dcc5c11c-f607-4a2e-bdaf-988dd9416309-kube-api-access-kb82b\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:34:44.049596 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:44.049584    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcc5c11c-f607-4a2e-bdaf-988dd9416309-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:34:44.051528 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:44.051482    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcc5c11c-f607-4a2e-bdaf-988dd9416309-webhook-cert" pod "dcc5c11c-f607-4a2e-bdaf-988dd9416309" (UID: "dcc5c11c-f607-4a2e-bdaf-988dd9416309"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:34:44.150013 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:44.149985    4327 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dcc5c11c-f607-4a2e-bdaf-988dd9416309-webhook-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:34:44.564794 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:44.564761    4327 generic.go:356] "Generic (PLEG): container finished" podID="dcc5c11c-f607-4a2e-bdaf-988dd9416309" containerID="70b9fefd9301243962826061b1ad761999642826c7e492e777834bacb18e158e" exitCode=0
Oct 31 00:34:44.565200 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:44.564801    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-2fksl" event={"ID":"dcc5c11c-f607-4a2e-bdaf-988dd9416309","Type":"ContainerDied","Data":"70b9fefd9301243962826061b1ad761999642826c7e492e777834bacb18e158e"}
Oct 31 00:34:44.565200 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:44.564840    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-2fksl" event={"ID":"dcc5c11c-f607-4a2e-bdaf-988dd9416309","Type":"ContainerDied","Data":"b4c6f1452bd0502ba6cb5a3e73ef1eafc8a6ddfdc775088e569285649a20c7bb"}
Oct 31 00:34:44.565200 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:44.564861    4327 scope.go:122] "RemoveContainer" containerID="70b9fefd9301243962826061b1ad761999642826c7e492e777834bacb18e158e"
Oct 31 00:34:44.565200 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:44.564806    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-2fksl"
Oct 31 00:34:44.573483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:44.573464    4327 scope.go:122] "RemoveContainer" containerID="70b9fefd9301243962826061b1ad761999642826c7e492e777834bacb18e158e"
Oct 31 00:34:44.573706 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:34:44.573670    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70b9fefd9301243962826061b1ad761999642826c7e492e777834bacb18e158e\": container with ID starting with 70b9fefd9301243962826061b1ad761999642826c7e492e777834bacb18e158e not found: ID does not exist" containerID="70b9fefd9301243962826061b1ad761999642826c7e492e777834bacb18e158e"
Oct 31 00:34:44.573779 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:44.573721    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70b9fefd9301243962826061b1ad761999642826c7e492e777834bacb18e158e"} err="failed to get container status \"70b9fefd9301243962826061b1ad761999642826c7e492e777834bacb18e158e\": rpc error: code = NotFound desc = could not find container \"70b9fefd9301243962826061b1ad761999642826c7e492e777834bacb18e158e\": container with ID starting with 70b9fefd9301243962826061b1ad761999642826c7e492e777834bacb18e158e not found: ID does not exist"
Oct 31 00:34:44.615962 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:44.615919    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-2fksl"]
Oct 31 00:34:44.628236 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:44.628204    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-2fksl"]
Oct 31 00:34:45.290886 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:34:45.290851    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:34:45.923113 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:34:45.919759    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcc5c11c-f607-4a2e-bdaf-988dd9416309" path="/var/lib/kubelet/pods/dcc5c11c-f607-4a2e-bdaf-988dd9416309/volumes"
Oct 31 00:35:07.140547 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:07.140510    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb"]
Oct 31 00:35:07.141004 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:07.140815    4327 cpu_manager.go:414] "containerMap: removing container" podUID="dcc5c11c-f607-4a2e-bdaf-988dd9416309" containerName="manager"
Oct 31 00:35:07.141004 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:07.140826    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcc5c11c-f607-4a2e-bdaf-988dd9416309" containerName="manager"
Oct 31 00:35:07.141004 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:07.140887    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcc5c11c-f607-4a2e-bdaf-988dd9416309" containerName="manager"
Oct 31 00:35:07.146923 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:07.146904    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb"
Oct 31 00:35:07.177102 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:07.177073    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 31 00:35:07.177102 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:07.177100    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cckh7\""
Oct 31 00:35:07.177362 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:07.177108    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 00:35:07.178528 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:07.178511    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb"]
Oct 31 00:35:07.203817 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:07.203792    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/872164e6-e8e9-401a-9f31-13bbdc2195f4-metrics-certs\") pod \"maas-api-6d669c7fcd-k6vzb\" (UID: \"872164e6-e8e9-401a-9f31-13bbdc2195f4\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb"
Oct 31 00:35:07.203937 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:07.203828    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/872164e6-e8e9-401a-9f31-13bbdc2195f4-maas-api-tls\") pod \"maas-api-6d669c7fcd-k6vzb\" (UID: \"872164e6-e8e9-401a-9f31-13bbdc2195f4\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb"
Oct 31 00:35:07.203937 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:07.203909    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdc6n\" (UniqueName: \"kubernetes.io/projected/872164e6-e8e9-401a-9f31-13bbdc2195f4-kube-api-access-xdc6n\") pod \"maas-api-6d669c7fcd-k6vzb\" (UID: \"872164e6-e8e9-401a-9f31-13bbdc2195f4\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb"
Oct 31 00:35:07.204038 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:07.204017    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/872164e6-e8e9-401a-9f31-13bbdc2195f4-openshift-service-ca\") pod \"maas-api-6d669c7fcd-k6vzb\" (UID: \"872164e6-e8e9-401a-9f31-13bbdc2195f4\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb"
Oct 31 00:35:07.304934 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:07.304889    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/872164e6-e8e9-401a-9f31-13bbdc2195f4-openshift-service-ca\") pod \"maas-api-6d669c7fcd-k6vzb\" (UID: \"872164e6-e8e9-401a-9f31-13bbdc2195f4\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb"
Oct 31 00:35:07.305118 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:07.304955    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/872164e6-e8e9-401a-9f31-13bbdc2195f4-metrics-certs\") pod \"maas-api-6d669c7fcd-k6vzb\" (UID: \"872164e6-e8e9-401a-9f31-13bbdc2195f4\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb"
Oct 31 00:35:07.305118 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:07.304990    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/872164e6-e8e9-401a-9f31-13bbdc2195f4-maas-api-tls\") pod \"maas-api-6d669c7fcd-k6vzb\" (UID: \"872164e6-e8e9-401a-9f31-13bbdc2195f4\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb"
Oct 31 00:35:07.305118 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:07.305063    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdc6n\" (UniqueName: \"kubernetes.io/projected/872164e6-e8e9-401a-9f31-13bbdc2195f4-kube-api-access-xdc6n\") pod \"maas-api-6d669c7fcd-k6vzb\" (UID: \"872164e6-e8e9-401a-9f31-13bbdc2195f4\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb"
Oct 31 00:35:07.305555 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:07.305534    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/872164e6-e8e9-401a-9f31-13bbdc2195f4-openshift-service-ca\") pod \"maas-api-6d669c7fcd-k6vzb\" (UID: \"872164e6-e8e9-401a-9f31-13bbdc2195f4\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb"
Oct 31 00:35:07.307472 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:07.307444    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/872164e6-e8e9-401a-9f31-13bbdc2195f4-metrics-certs\") pod \"maas-api-6d669c7fcd-k6vzb\" (UID: \"872164e6-e8e9-401a-9f31-13bbdc2195f4\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb"
Oct 31 00:35:07.307472 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:07.307469    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/872164e6-e8e9-401a-9f31-13bbdc2195f4-maas-api-tls\") pod \"maas-api-6d669c7fcd-k6vzb\" (UID: \"872164e6-e8e9-401a-9f31-13bbdc2195f4\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb"
Oct 31 00:35:07.330691 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:07.330647    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdc6n\" (UniqueName: \"kubernetes.io/projected/872164e6-e8e9-401a-9f31-13bbdc2195f4-kube-api-access-xdc6n\") pod \"maas-api-6d669c7fcd-k6vzb\" (UID: \"872164e6-e8e9-401a-9f31-13bbdc2195f4\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb"
Oct 31 00:35:07.458923 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:07.458844    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb"
Oct 31 00:35:07.758038 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:07.758007    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb"]
Oct 31 00:35:07.758818 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:35:07.758785    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872164e6_e8e9_401a_9f31_13bbdc2195f4.slice/crio-a925938e0f7d04c9beb9c02ef46dac23ad114c72005c40308b58fa93abf98263 WatchSource:0}: Error finding container a925938e0f7d04c9beb9c02ef46dac23ad114c72005c40308b58fa93abf98263: Status 404 returned error can't find the container with id a925938e0f7d04c9beb9c02ef46dac23ad114c72005c40308b58fa93abf98263
Oct 31 00:35:08.661741 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:08.661698    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb" event={"ID":"872164e6-e8e9-401a-9f31-13bbdc2195f4","Type":"ContainerStarted","Data":"a925938e0f7d04c9beb9c02ef46dac23ad114c72005c40308b58fa93abf98263"}
Oct 31 00:35:09.667199 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:09.667108    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb" event={"ID":"872164e6-e8e9-401a-9f31-13bbdc2195f4","Type":"ContainerStarted","Data":"0036eb45de5b0e15f36f4e9dae95ef0f4b05c8c1df71394c3ba7a8ffde6b1551"}
Oct 31 00:35:09.667586 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:09.667229    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb"
Oct 31 00:35:09.712804 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:09.712750    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb" podStartSLOduration=1.082994322 podStartE2EDuration="2.71273414s" totalImagesPullingTime="1.629739818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:09.711354838 +0000 UTC m=+969.933390380" watchObservedRunningTime="2026-10-31 00:35:09.71273414 +0000 UTC m=+969.934769682"
Oct 31 00:35:15.291287 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:35:15.291245    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:35:15.679352 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:15.679320    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb"
Oct 31 00:35:16.647565 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.647529    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8"]
Oct 31 00:35:16.650837 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.650817    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8"
Oct 31 00:35:16.666642 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.666618    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de8f47da-a2b5-4b42-825c-fd08144e9028-maas-api-tls\") pod \"maas-api-6fb4ff9887-splg8\" (UID: \"de8f47da-a2b5-4b42-825c-fd08144e9028\") " pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8"
Oct 31 00:35:16.666782 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.666696    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de8f47da-a2b5-4b42-825c-fd08144e9028-openshift-service-ca\") pod \"maas-api-6fb4ff9887-splg8\" (UID: \"de8f47da-a2b5-4b42-825c-fd08144e9028\") " pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8"
Oct 31 00:35:16.666782 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.666768    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssjml\" (UniqueName: \"kubernetes.io/projected/de8f47da-a2b5-4b42-825c-fd08144e9028-kube-api-access-ssjml\") pod \"maas-api-6fb4ff9887-splg8\" (UID: \"de8f47da-a2b5-4b42-825c-fd08144e9028\") " pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8"
Oct 31 00:35:16.666860 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.666805    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de8f47da-a2b5-4b42-825c-fd08144e9028-metrics-certs\") pod \"maas-api-6fb4ff9887-splg8\" (UID: \"de8f47da-a2b5-4b42-825c-fd08144e9028\") " pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8"
Oct 31 00:35:16.681101 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.681075    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8"]
Oct 31 00:35:16.682945 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.682924    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-6mfzt"]
Oct 31 00:35:16.686077 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.686062    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-6mfzt"
Oct 31 00:35:16.703598 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.703362    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 31 00:35:16.704622 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.704594    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-6mfzt"]
Oct 31 00:35:16.767856 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.767812    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cc6c5bd2-557b-4476-bb78-cbda3be0f0b5-tls-cert\") pod \"authorino-6889cbd8c4-6mfzt\" (UID: \"cc6c5bd2-557b-4476-bb78-cbda3be0f0b5\") " pod="kuadrant-system/authorino-6889cbd8c4-6mfzt"
Oct 31 00:35:16.768037 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.767887    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de8f47da-a2b5-4b42-825c-fd08144e9028-openshift-service-ca\") pod \"maas-api-6fb4ff9887-splg8\" (UID: \"de8f47da-a2b5-4b42-825c-fd08144e9028\") " pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8"
Oct 31 00:35:16.768037 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.767928    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl44b\" (UniqueName: \"kubernetes.io/projected/cc6c5bd2-557b-4476-bb78-cbda3be0f0b5-kube-api-access-bl44b\") pod \"authorino-6889cbd8c4-6mfzt\" (UID: \"cc6c5bd2-557b-4476-bb78-cbda3be0f0b5\") " pod="kuadrant-system/authorino-6889cbd8c4-6mfzt"
Oct 31 00:35:16.768037 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.768031    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssjml\" (UniqueName: \"kubernetes.io/projected/de8f47da-a2b5-4b42-825c-fd08144e9028-kube-api-access-ssjml\") pod \"maas-api-6fb4ff9887-splg8\" (UID: \"de8f47da-a2b5-4b42-825c-fd08144e9028\") " pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8"
Oct 31 00:35:16.768210 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.768073    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de8f47da-a2b5-4b42-825c-fd08144e9028-metrics-certs\") pod \"maas-api-6fb4ff9887-splg8\" (UID: \"de8f47da-a2b5-4b42-825c-fd08144e9028\") " pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8"
Oct 31 00:35:16.768210 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.768113    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de8f47da-a2b5-4b42-825c-fd08144e9028-maas-api-tls\") pod \"maas-api-6fb4ff9887-splg8\" (UID: \"de8f47da-a2b5-4b42-825c-fd08144e9028\") " pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8"
Oct 31 00:35:16.768618 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.768597    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de8f47da-a2b5-4b42-825c-fd08144e9028-openshift-service-ca\") pod \"maas-api-6fb4ff9887-splg8\" (UID: \"de8f47da-a2b5-4b42-825c-fd08144e9028\") " pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8"
Oct 31 00:35:16.770537 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.770512    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de8f47da-a2b5-4b42-825c-fd08144e9028-maas-api-tls\") pod \"maas-api-6fb4ff9887-splg8\" (UID: \"de8f47da-a2b5-4b42-825c-fd08144e9028\") " pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8"
Oct 31 00:35:16.770632 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.770590    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de8f47da-a2b5-4b42-825c-fd08144e9028-metrics-certs\") pod \"maas-api-6fb4ff9887-splg8\" (UID: \"de8f47da-a2b5-4b42-825c-fd08144e9028\") " pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8"
Oct 31 00:35:16.793106 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.793083    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssjml\" (UniqueName: \"kubernetes.io/projected/de8f47da-a2b5-4b42-825c-fd08144e9028-kube-api-access-ssjml\") pod \"maas-api-6fb4ff9887-splg8\" (UID: \"de8f47da-a2b5-4b42-825c-fd08144e9028\") " pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8"
Oct 31 00:35:16.820209 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.820176    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-6mfzt"]
Oct 31 00:35:16.820403 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:35:16.820384    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-bl44b tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-6mfzt" podUID="cc6c5bd2-557b-4476-bb78-cbda3be0f0b5"
Oct 31 00:35:16.869226 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.869193    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl44b\" (UniqueName: \"kubernetes.io/projected/cc6c5bd2-557b-4476-bb78-cbda3be0f0b5-kube-api-access-bl44b\") pod \"authorino-6889cbd8c4-6mfzt\" (UID: \"cc6c5bd2-557b-4476-bb78-cbda3be0f0b5\") " pod="kuadrant-system/authorino-6889cbd8c4-6mfzt"
Oct 31 00:35:16.869418 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.869291    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cc6c5bd2-557b-4476-bb78-cbda3be0f0b5-tls-cert\") pod \"authorino-6889cbd8c4-6mfzt\" (UID: \"cc6c5bd2-557b-4476-bb78-cbda3be0f0b5\") " pod="kuadrant-system/authorino-6889cbd8c4-6mfzt"
Oct 31 00:35:16.871608 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.871581    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cc6c5bd2-557b-4476-bb78-cbda3be0f0b5-tls-cert\") pod \"authorino-6889cbd8c4-6mfzt\" (UID: \"cc6c5bd2-557b-4476-bb78-cbda3be0f0b5\") " pod="kuadrant-system/authorino-6889cbd8c4-6mfzt"
Oct 31 00:35:16.893279 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.893255    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl44b\" (UniqueName: \"kubernetes.io/projected/cc6c5bd2-557b-4476-bb78-cbda3be0f0b5-kube-api-access-bl44b\") pod \"authorino-6889cbd8c4-6mfzt\" (UID: \"cc6c5bd2-557b-4476-bb78-cbda3be0f0b5\") " pod="kuadrant-system/authorino-6889cbd8c4-6mfzt"
Oct 31 00:35:16.964058 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:16.963973    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8"
Oct 31 00:35:17.258667 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:17.258634    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8"]
Oct 31 00:35:17.260023 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:35:17.259999    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde8f47da_a2b5_4b42_825c_fd08144e9028.slice/crio-af0e134fa6a67c2046c64b14cacfb33f692ef95d5f09621d546aab9dbc25b39a WatchSource:0}: Error finding container af0e134fa6a67c2046c64b14cacfb33f692ef95d5f09621d546aab9dbc25b39a: Status 404 returned error can't find the container with id af0e134fa6a67c2046c64b14cacfb33f692ef95d5f09621d546aab9dbc25b39a
Oct 31 00:35:17.703227 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:17.703191    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8" event={"ID":"de8f47da-a2b5-4b42-825c-fd08144e9028","Type":"ContainerStarted","Data":"232b23d8c027effd696dcd5800809cff4515ea3f16c2a0023866fd0ea39eceec"}
Oct 31 00:35:17.703631 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:17.703264    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-6mfzt"
Oct 31 00:35:17.703631 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:17.703287    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8" event={"ID":"de8f47da-a2b5-4b42-825c-fd08144e9028","Type":"ContainerStarted","Data":"af0e134fa6a67c2046c64b14cacfb33f692ef95d5f09621d546aab9dbc25b39a"}
Oct 31 00:35:17.703631 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:17.703321    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8"
Oct 31 00:35:17.709674 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:17.709655    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-6mfzt"
Oct 31 00:35:17.774313 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:17.774280    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc6c5bd2-557b-4476-bb78-cbda3be0f0b5-tls-cert\" (UniqueName: \"kubernetes.io/secret/cc6c5bd2-557b-4476-bb78-cbda3be0f0b5-tls-cert\") pod \"cc6c5bd2-557b-4476-bb78-cbda3be0f0b5\" (UID: \"cc6c5bd2-557b-4476-bb78-cbda3be0f0b5\") "
Oct 31 00:35:17.774477 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:17.774338    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc6c5bd2-557b-4476-bb78-cbda3be0f0b5-kube-api-access-bl44b\" (UniqueName: \"kubernetes.io/projected/cc6c5bd2-557b-4476-bb78-cbda3be0f0b5-kube-api-access-bl44b\") pod \"cc6c5bd2-557b-4476-bb78-cbda3be0f0b5\" (UID: \"cc6c5bd2-557b-4476-bb78-cbda3be0f0b5\") "
Oct 31 00:35:17.776504 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:17.776473    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc6c5bd2-557b-4476-bb78-cbda3be0f0b5-tls-cert" pod "cc6c5bd2-557b-4476-bb78-cbda3be0f0b5" (UID: "cc6c5bd2-557b-4476-bb78-cbda3be0f0b5"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:35:17.776504 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:17.776496    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6c5bd2-557b-4476-bb78-cbda3be0f0b5-kube-api-access-bl44b" pod "cc6c5bd2-557b-4476-bb78-cbda3be0f0b5" (UID: "cc6c5bd2-557b-4476-bb78-cbda3be0f0b5"). InnerVolumeSpecName "kube-api-access-bl44b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:35:17.785702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:17.785640    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8" podStartSLOduration=1.422528145 podStartE2EDuration="1.785623121s" totalImagesPullingTime="363.094976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:17.784077382 +0000 UTC m=+978.006112926" watchObservedRunningTime="2026-10-31 00:35:17.785623121 +0000 UTC m=+978.007658664"
Oct 31 00:35:17.875184 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:17.875102    4327 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cc6c5bd2-557b-4476-bb78-cbda3be0f0b5-tls-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:35:17.875184 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:17.875131    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl44b\" (UniqueName: \"kubernetes.io/projected/cc6c5bd2-557b-4476-bb78-cbda3be0f0b5-kube-api-access-bl44b\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:35:18.707475 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:18.707444    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-6mfzt"
Oct 31 00:35:18.785230 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:18.785196    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-6mfzt"]
Oct 31 00:35:18.791851 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:18.791814    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-6mfzt"]
Oct 31 00:35:19.919363 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:19.919324    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc6c5bd2-557b-4476-bb78-cbda3be0f0b5" path="/var/lib/kubelet/pods/cc6c5bd2-557b-4476-bb78-cbda3be0f0b5/volumes"
Oct 31 00:35:23.712911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:23.712880    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8"
Oct 31 00:35:23.788886 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:23.788853    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb"]
Oct 31 00:35:23.789094 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:23.789069    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb" podUID="872164e6-e8e9-401a-9f31-13bbdc2195f4" containerName="maas-api" containerID="cri-o://0036eb45de5b0e15f36f4e9dae95ef0f4b05c8c1df71394c3ba7a8ffde6b1551" gracePeriod=30
Oct 31 00:35:24.042619 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:24.042594    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb"
Oct 31 00:35:24.124799 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:24.124767    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/872164e6-e8e9-401a-9f31-13bbdc2195f4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/872164e6-e8e9-401a-9f31-13bbdc2195f4-maas-api-tls\") pod \"872164e6-e8e9-401a-9f31-13bbdc2195f4\" (UID: \"872164e6-e8e9-401a-9f31-13bbdc2195f4\") "
Oct 31 00:35:24.124973 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:24.124809    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/872164e6-e8e9-401a-9f31-13bbdc2195f4-kube-api-access-xdc6n\" (UniqueName: \"kubernetes.io/projected/872164e6-e8e9-401a-9f31-13bbdc2195f4-kube-api-access-xdc6n\") pod \"872164e6-e8e9-401a-9f31-13bbdc2195f4\" (UID: \"872164e6-e8e9-401a-9f31-13bbdc2195f4\") "
Oct 31 00:35:24.124973 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:24.124847    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/872164e6-e8e9-401a-9f31-13bbdc2195f4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/872164e6-e8e9-401a-9f31-13bbdc2195f4-openshift-service-ca\") pod \"872164e6-e8e9-401a-9f31-13bbdc2195f4\" (UID: \"872164e6-e8e9-401a-9f31-13bbdc2195f4\") "
Oct 31 00:35:24.124973 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:24.124870    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/872164e6-e8e9-401a-9f31-13bbdc2195f4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/872164e6-e8e9-401a-9f31-13bbdc2195f4-metrics-certs\") pod \"872164e6-e8e9-401a-9f31-13bbdc2195f4\" (UID: \"872164e6-e8e9-401a-9f31-13bbdc2195f4\") "
Oct 31 00:35:24.125257 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:24.125228    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/872164e6-e8e9-401a-9f31-13bbdc2195f4-openshift-service-ca" pod "872164e6-e8e9-401a-9f31-13bbdc2195f4" (UID: "872164e6-e8e9-401a-9f31-13bbdc2195f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:35:24.127018 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:24.126986    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/872164e6-e8e9-401a-9f31-13bbdc2195f4-maas-api-tls" pod "872164e6-e8e9-401a-9f31-13bbdc2195f4" (UID: "872164e6-e8e9-401a-9f31-13bbdc2195f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:35:24.127119 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:24.127012    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/872164e6-e8e9-401a-9f31-13bbdc2195f4-metrics-certs" pod "872164e6-e8e9-401a-9f31-13bbdc2195f4" (UID: "872164e6-e8e9-401a-9f31-13bbdc2195f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:35:24.127119 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:24.127039    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872164e6-e8e9-401a-9f31-13bbdc2195f4-kube-api-access-xdc6n" pod "872164e6-e8e9-401a-9f31-13bbdc2195f4" (UID: "872164e6-e8e9-401a-9f31-13bbdc2195f4"). InnerVolumeSpecName "kube-api-access-xdc6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:35:24.225942 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:24.225911    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/872164e6-e8e9-401a-9f31-13bbdc2195f4-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:35:24.225942 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:24.225936    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdc6n\" (UniqueName: \"kubernetes.io/projected/872164e6-e8e9-401a-9f31-13bbdc2195f4-kube-api-access-xdc6n\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:35:24.225942 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:24.225945    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/872164e6-e8e9-401a-9f31-13bbdc2195f4-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:35:24.226146 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:24.225954    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/872164e6-e8e9-401a-9f31-13bbdc2195f4-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:35:24.733793 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:24.733759    4327 generic.go:356] "Generic (PLEG): container finished" podID="872164e6-e8e9-401a-9f31-13bbdc2195f4" containerID="0036eb45de5b0e15f36f4e9dae95ef0f4b05c8c1df71394c3ba7a8ffde6b1551" exitCode=0
Oct 31 00:35:24.734202 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:24.733805    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb"
Oct 31 00:35:24.734202 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:24.733814    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb" event={"ID":"872164e6-e8e9-401a-9f31-13bbdc2195f4","Type":"ContainerDied","Data":"0036eb45de5b0e15f36f4e9dae95ef0f4b05c8c1df71394c3ba7a8ffde6b1551"}
Oct 31 00:35:24.734202 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:24.733881    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb" event={"ID":"872164e6-e8e9-401a-9f31-13bbdc2195f4","Type":"ContainerDied","Data":"a925938e0f7d04c9beb9c02ef46dac23ad114c72005c40308b58fa93abf98263"}
Oct 31 00:35:24.734202 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:24.733904    4327 scope.go:122] "RemoveContainer" containerID="0036eb45de5b0e15f36f4e9dae95ef0f4b05c8c1df71394c3ba7a8ffde6b1551"
Oct 31 00:35:24.742635 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:24.742616    4327 scope.go:122] "RemoveContainer" containerID="0036eb45de5b0e15f36f4e9dae95ef0f4b05c8c1df71394c3ba7a8ffde6b1551"
Oct 31 00:35:24.742936 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:35:24.742916    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0036eb45de5b0e15f36f4e9dae95ef0f4b05c8c1df71394c3ba7a8ffde6b1551\": container with ID starting with 0036eb45de5b0e15f36f4e9dae95ef0f4b05c8c1df71394c3ba7a8ffde6b1551 not found: ID does not exist" containerID="0036eb45de5b0e15f36f4e9dae95ef0f4b05c8c1df71394c3ba7a8ffde6b1551"
Oct 31 00:35:24.742994 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:24.742947    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0036eb45de5b0e15f36f4e9dae95ef0f4b05c8c1df71394c3ba7a8ffde6b1551"} err="failed to get container status \"0036eb45de5b0e15f36f4e9dae95ef0f4b05c8c1df71394c3ba7a8ffde6b1551\": rpc error: code = NotFound desc = could not find container \"0036eb45de5b0e15f36f4e9dae95ef0f4b05c8c1df71394c3ba7a8ffde6b1551\": container with ID starting with 0036eb45de5b0e15f36f4e9dae95ef0f4b05c8c1df71394c3ba7a8ffde6b1551 not found: ID does not exist"
Oct 31 00:35:24.785330 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:24.785302    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb"]
Oct 31 00:35:24.797606 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:24.797575    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-k6vzb"]
Oct 31 00:35:25.918395 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:25.918363    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="872164e6-e8e9-401a-9f31-13bbdc2195f4" path="/var/lib/kubelet/pods/872164e6-e8e9-401a-9f31-13bbdc2195f4/volumes"
Oct 31 00:35:26.935907 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:26.935875    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nrpnv"]
Oct 31 00:35:26.936348 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:26.936095    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-nrpnv" podUID="9edd5747-b0b2-4d57-a74c-82e5a425a8ee" containerName="authorino" containerID="cri-o://1f0f9db77631b4f96f96263b8240fc213f96e7057dd880dd6c4f03d09ac294f9" gracePeriod=30
Oct 31 00:35:27.174503 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:27.174478    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nrpnv"
Oct 31 00:35:27.246385 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:27.246305    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9edd5747-b0b2-4d57-a74c-82e5a425a8ee-kube-api-access-7s9xt\" (UniqueName: \"kubernetes.io/projected/9edd5747-b0b2-4d57-a74c-82e5a425a8ee-kube-api-access-7s9xt\") pod \"9edd5747-b0b2-4d57-a74c-82e5a425a8ee\" (UID: \"9edd5747-b0b2-4d57-a74c-82e5a425a8ee\") "
Oct 31 00:35:27.248340 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:27.248307    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9edd5747-b0b2-4d57-a74c-82e5a425a8ee-kube-api-access-7s9xt" pod "9edd5747-b0b2-4d57-a74c-82e5a425a8ee" (UID: "9edd5747-b0b2-4d57-a74c-82e5a425a8ee"). InnerVolumeSpecName "kube-api-access-7s9xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:35:27.346986 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:27.346949    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s9xt\" (UniqueName: \"kubernetes.io/projected/9edd5747-b0b2-4d57-a74c-82e5a425a8ee-kube-api-access-7s9xt\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:35:27.747851 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:27.747807    4327 generic.go:356] "Generic (PLEG): container finished" podID="9edd5747-b0b2-4d57-a74c-82e5a425a8ee" containerID="1f0f9db77631b4f96f96263b8240fc213f96e7057dd880dd6c4f03d09ac294f9" exitCode=0
Oct 31 00:35:27.748029 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:27.747859    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nrpnv"
Oct 31 00:35:27.748029 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:27.747886    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nrpnv" event={"ID":"9edd5747-b0b2-4d57-a74c-82e5a425a8ee","Type":"ContainerDied","Data":"1f0f9db77631b4f96f96263b8240fc213f96e7057dd880dd6c4f03d09ac294f9"}
Oct 31 00:35:27.748029 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:27.747930    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nrpnv" event={"ID":"9edd5747-b0b2-4d57-a74c-82e5a425a8ee","Type":"ContainerDied","Data":"a00673b00ede3d7923719ea7320aeab4b223d670f56080ffdb363e097ddcd2cf"}
Oct 31 00:35:27.748029 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:27.747945    4327 scope.go:122] "RemoveContainer" containerID="1f0f9db77631b4f96f96263b8240fc213f96e7057dd880dd6c4f03d09ac294f9"
Oct 31 00:35:27.756635 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:27.756612    4327 scope.go:122] "RemoveContainer" containerID="1f0f9db77631b4f96f96263b8240fc213f96e7057dd880dd6c4f03d09ac294f9"
Oct 31 00:35:27.756943 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:35:27.756896    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f0f9db77631b4f96f96263b8240fc213f96e7057dd880dd6c4f03d09ac294f9\": container with ID starting with 1f0f9db77631b4f96f96263b8240fc213f96e7057dd880dd6c4f03d09ac294f9 not found: ID does not exist" containerID="1f0f9db77631b4f96f96263b8240fc213f96e7057dd880dd6c4f03d09ac294f9"
Oct 31 00:35:27.756943 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:27.756924    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f0f9db77631b4f96f96263b8240fc213f96e7057dd880dd6c4f03d09ac294f9"} err="failed to get container status \"1f0f9db77631b4f96f96263b8240fc213f96e7057dd880dd6c4f03d09ac294f9\": rpc error: code = NotFound desc = could not find container \"1f0f9db77631b4f96f96263b8240fc213f96e7057dd880dd6c4f03d09ac294f9\": container with ID starting with 1f0f9db77631b4f96f96263b8240fc213f96e7057dd880dd6c4f03d09ac294f9 not found: ID does not exist"
Oct 31 00:35:27.798753 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:27.798714    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nrpnv"]
Oct 31 00:35:27.804006 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:27.803982    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nrpnv"]
Oct 31 00:35:27.918633 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:35:27.918600    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9edd5747-b0b2-4d57-a74c-82e5a425a8ee" path="/var/lib/kubelet/pods/9edd5747-b0b2-4d57-a74c-82e5a425a8ee/volumes"
Oct 31 00:35:45.290674 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:35:45.290631    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:36:15.291638 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:36:15.291556    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:36:19.668358 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.668325    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-dddkj"]
Oct 31 00:36:19.668742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.668620    4327 cpu_manager.go:414] "containerMap: removing container" podUID="9edd5747-b0b2-4d57-a74c-82e5a425a8ee" containerName="authorino"
Oct 31 00:36:19.668742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.668630    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9edd5747-b0b2-4d57-a74c-82e5a425a8ee" containerName="authorino"
Oct 31 00:36:19.668742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.668647    4327 cpu_manager.go:414] "containerMap: removing container" podUID="872164e6-e8e9-401a-9f31-13bbdc2195f4" containerName="maas-api"
Oct 31 00:36:19.668742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.668652    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="872164e6-e8e9-401a-9f31-13bbdc2195f4" containerName="maas-api"
Oct 31 00:36:19.668742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.668719    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="9edd5747-b0b2-4d57-a74c-82e5a425a8ee" containerName="authorino"
Oct 31 00:36:19.668742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.668730    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="872164e6-e8e9-401a-9f31-13bbdc2195f4" containerName="maas-api"
Oct 31 00:36:19.671783 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.671755    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj"
Oct 31 00:36:19.677875 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.677839    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1988c46-0b7f-4edd-b441-ab408ad0a246-home\") pod \"echo-simulator-kserve-64b6f8f96c-dddkj\" (UID: \"b1988c46-0b7f-4edd-b441-ab408ad0a246\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj"
Oct 31 00:36:19.678389 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.677885    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1988c46-0b7f-4edd-b441-ab408ad0a246-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-dddkj\" (UID: \"b1988c46-0b7f-4edd-b441-ab408ad0a246\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj"
Oct 31 00:36:19.678389 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.677936    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh6g7\" (UniqueName: \"kubernetes.io/projected/b1988c46-0b7f-4edd-b441-ab408ad0a246-kube-api-access-qh6g7\") pod \"echo-simulator-kserve-64b6f8f96c-dddkj\" (UID: \"b1988c46-0b7f-4edd-b441-ab408ad0a246\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj"
Oct 31 00:36:19.678389 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.678052    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1988c46-0b7f-4edd-b441-ab408ad0a246-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-dddkj\" (UID: \"b1988c46-0b7f-4edd-b441-ab408ad0a246\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj"
Oct 31 00:36:19.678389 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.678100    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1988c46-0b7f-4edd-b441-ab408ad0a246-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-dddkj\" (UID: \"b1988c46-0b7f-4edd-b441-ab408ad0a246\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj"
Oct 31 00:36:19.678389 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.678150    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1988c46-0b7f-4edd-b441-ab408ad0a246-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-dddkj\" (UID: \"b1988c46-0b7f-4edd-b441-ab408ad0a246\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj"
Oct 31 00:36:19.699068 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.699044    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 31 00:36:19.699068 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.699065    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 31 00:36:19.699258 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.699068    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-gdwmn\""
Oct 31 00:36:19.699258 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.699066    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 31 00:36:19.700336 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.700319    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-dddkj"]
Oct 31 00:36:19.778856 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.778818    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1988c46-0b7f-4edd-b441-ab408ad0a246-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-dddkj\" (UID: \"b1988c46-0b7f-4edd-b441-ab408ad0a246\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj"
Oct 31 00:36:19.779014 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.778868    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1988c46-0b7f-4edd-b441-ab408ad0a246-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-dddkj\" (UID: \"b1988c46-0b7f-4edd-b441-ab408ad0a246\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj"
Oct 31 00:36:19.779014 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.778908    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1988c46-0b7f-4edd-b441-ab408ad0a246-home\") pod \"echo-simulator-kserve-64b6f8f96c-dddkj\" (UID: \"b1988c46-0b7f-4edd-b441-ab408ad0a246\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj"
Oct 31 00:36:19.779014 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.778937    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1988c46-0b7f-4edd-b441-ab408ad0a246-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-dddkj\" (UID: \"b1988c46-0b7f-4edd-b441-ab408ad0a246\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj"
Oct 31 00:36:19.779014 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.778956    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh6g7\" (UniqueName: \"kubernetes.io/projected/b1988c46-0b7f-4edd-b441-ab408ad0a246-kube-api-access-qh6g7\") pod \"echo-simulator-kserve-64b6f8f96c-dddkj\" (UID: \"b1988c46-0b7f-4edd-b441-ab408ad0a246\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj"
Oct 31 00:36:19.779014 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.779012    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1988c46-0b7f-4edd-b441-ab408ad0a246-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-dddkj\" (UID: \"b1988c46-0b7f-4edd-b441-ab408ad0a246\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj"
Oct 31 00:36:19.779300 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.779280    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1988c46-0b7f-4edd-b441-ab408ad0a246-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-dddkj\" (UID: \"b1988c46-0b7f-4edd-b441-ab408ad0a246\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj"
Oct 31 00:36:19.779358 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.779321    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1988c46-0b7f-4edd-b441-ab408ad0a246-home\") pod \"echo-simulator-kserve-64b6f8f96c-dddkj\" (UID: \"b1988c46-0b7f-4edd-b441-ab408ad0a246\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj"
Oct 31 00:36:19.779358 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.779343    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1988c46-0b7f-4edd-b441-ab408ad0a246-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-dddkj\" (UID: \"b1988c46-0b7f-4edd-b441-ab408ad0a246\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj"
Oct 31 00:36:19.781325 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.781295    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1988c46-0b7f-4edd-b441-ab408ad0a246-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-dddkj\" (UID: \"b1988c46-0b7f-4edd-b441-ab408ad0a246\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj"
Oct 31 00:36:19.781483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.781466    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1988c46-0b7f-4edd-b441-ab408ad0a246-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-dddkj\" (UID: \"b1988c46-0b7f-4edd-b441-ab408ad0a246\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj"
Oct 31 00:36:19.803354 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.803319    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh6g7\" (UniqueName: \"kubernetes.io/projected/b1988c46-0b7f-4edd-b441-ab408ad0a246-kube-api-access-qh6g7\") pod \"echo-simulator-kserve-64b6f8f96c-dddkj\" (UID: \"b1988c46-0b7f-4edd-b441-ab408ad0a246\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj"
Oct 31 00:36:19.984273 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:19.984171    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj"
Oct 31 00:36:20.285407 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:20.285382    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-dddkj"]
Oct 31 00:36:20.286345 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:36:20.286316    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1988c46_0b7f_4edd_b441_ab408ad0a246.slice/crio-78fe3ff1b38b594808f10c29798c5edd71749bcd62936568f66f209ffd3f2f46 WatchSource:0}: Error finding container 78fe3ff1b38b594808f10c29798c5edd71749bcd62936568f66f209ffd3f2f46: Status 404 returned error can't find the container with id 78fe3ff1b38b594808f10c29798c5edd71749bcd62936568f66f209ffd3f2f46
Oct 31 00:36:20.958216 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:20.958141    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj" event={"ID":"b1988c46-0b7f-4edd-b441-ab408ad0a246","Type":"ContainerStarted","Data":"78fe3ff1b38b594808f10c29798c5edd71749bcd62936568f66f209ffd3f2f46"}
Oct 31 00:36:21.805587 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:21.805554    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"]
Oct 31 00:36:21.808897 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:21.808870    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"
Oct 31 00:36:21.838587 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:21.838554    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"]
Oct 31 00:36:21.838895 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:21.838856    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 31 00:36:21.895014 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:21.894978    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1398a6a-1ac9-48e1-a988-19c575e0cb91-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt\" (UID: \"c1398a6a-1ac9-48e1-a988-19c575e0cb91\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"
Oct 31 00:36:21.895014 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:21.895024    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1398a6a-1ac9-48e1-a988-19c575e0cb91-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt\" (UID: \"c1398a6a-1ac9-48e1-a988-19c575e0cb91\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"
Oct 31 00:36:21.895217 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:21.895123    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1398a6a-1ac9-48e1-a988-19c575e0cb91-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt\" (UID: \"c1398a6a-1ac9-48e1-a988-19c575e0cb91\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"
Oct 31 00:36:21.895217 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:21.895159    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1398a6a-1ac9-48e1-a988-19c575e0cb91-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt\" (UID: \"c1398a6a-1ac9-48e1-a988-19c575e0cb91\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"
Oct 31 00:36:21.895217 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:21.895178    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd62s\" (UniqueName: \"kubernetes.io/projected/c1398a6a-1ac9-48e1-a988-19c575e0cb91-kube-api-access-qd62s\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt\" (UID: \"c1398a6a-1ac9-48e1-a988-19c575e0cb91\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"
Oct 31 00:36:21.895313 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:21.895247    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1398a6a-1ac9-48e1-a988-19c575e0cb91-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt\" (UID: \"c1398a6a-1ac9-48e1-a988-19c575e0cb91\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"
Oct 31 00:36:21.963147 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:21.963068    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj" event={"ID":"b1988c46-0b7f-4edd-b441-ab408ad0a246","Type":"ContainerStarted","Data":"780f6203011481d695f0ba336510900d7f77e933b61dff04230c5c54294d1d9b"}
Oct 31 00:36:21.985032 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:21.985005    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj"
Oct 31 00:36:21.995771 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:21.995748    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1398a6a-1ac9-48e1-a988-19c575e0cb91-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt\" (UID: \"c1398a6a-1ac9-48e1-a988-19c575e0cb91\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"
Oct 31 00:36:21.995925 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:21.995789    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1398a6a-1ac9-48e1-a988-19c575e0cb91-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt\" (UID: \"c1398a6a-1ac9-48e1-a988-19c575e0cb91\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"
Oct 31 00:36:21.995994 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:21.995932    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd62s\" (UniqueName: \"kubernetes.io/projected/c1398a6a-1ac9-48e1-a988-19c575e0cb91-kube-api-access-qd62s\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt\" (UID: \"c1398a6a-1ac9-48e1-a988-19c575e0cb91\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"
Oct 31 00:36:21.996052 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:21.995991    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1398a6a-1ac9-48e1-a988-19c575e0cb91-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt\" (UID: \"c1398a6a-1ac9-48e1-a988-19c575e0cb91\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"
Oct 31 00:36:21.996107 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:21.996072    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1398a6a-1ac9-48e1-a988-19c575e0cb91-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt\" (UID: \"c1398a6a-1ac9-48e1-a988-19c575e0cb91\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"
Oct 31 00:36:21.996159 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:21.996116    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1398a6a-1ac9-48e1-a988-19c575e0cb91-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt\" (UID: \"c1398a6a-1ac9-48e1-a988-19c575e0cb91\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"
Oct 31 00:36:21.996159 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:21.996140    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1398a6a-1ac9-48e1-a988-19c575e0cb91-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt\" (UID: \"c1398a6a-1ac9-48e1-a988-19c575e0cb91\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"
Oct 31 00:36:21.996343 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:21.996208    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1398a6a-1ac9-48e1-a988-19c575e0cb91-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt\" (UID: \"c1398a6a-1ac9-48e1-a988-19c575e0cb91\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"
Oct 31 00:36:21.996343 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:21.996316    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1398a6a-1ac9-48e1-a988-19c575e0cb91-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt\" (UID: \"c1398a6a-1ac9-48e1-a988-19c575e0cb91\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"
Oct 31 00:36:21.998388 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:21.998368    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1398a6a-1ac9-48e1-a988-19c575e0cb91-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt\" (UID: \"c1398a6a-1ac9-48e1-a988-19c575e0cb91\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"
Oct 31 00:36:21.998586 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:21.998569    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1398a6a-1ac9-48e1-a988-19c575e0cb91-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt\" (UID: \"c1398a6a-1ac9-48e1-a988-19c575e0cb91\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"
Oct 31 00:36:22.009636 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:22.009589    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj" podStartSLOduration=1.596921402 podStartE2EDuration="3.009576366s" totalImagesPullingTime="1.412654964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:22.009110468 +0000 UTC m=+1042.231146011" watchObservedRunningTime="2026-10-31 00:36:22.009576366 +0000 UTC m=+1042.231611908"
Oct 31 00:36:22.020631 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:22.020605    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd62s\" (UniqueName: \"kubernetes.io/projected/c1398a6a-1ac9-48e1-a988-19c575e0cb91-kube-api-access-qd62s\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt\" (UID: \"c1398a6a-1ac9-48e1-a988-19c575e0cb91\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"
Oct 31 00:36:22.125946 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:22.125911    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"
Oct 31 00:36:22.422461 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:22.422432    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"]
Oct 31 00:36:22.422886 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:36:22.422859    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1398a6a_1ac9_48e1_a988_19c575e0cb91.slice/crio-694a74f4bf1450408b4bba5d574808215e57dc521b3d211585e928af3c5865bc WatchSource:0}: Error finding container 694a74f4bf1450408b4bba5d574808215e57dc521b3d211585e928af3c5865bc: Status 404 returned error can't find the container with id 694a74f4bf1450408b4bba5d574808215e57dc521b3d211585e928af3c5865bc
Oct 31 00:36:22.970443 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:22.970401    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt" event={"ID":"c1398a6a-1ac9-48e1-a988-19c575e0cb91","Type":"ContainerStarted","Data":"694a74f4bf1450408b4bba5d574808215e57dc521b3d211585e928af3c5865bc"}
Oct 31 00:36:23.978973 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:23.977825    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt" event={"ID":"c1398a6a-1ac9-48e1-a988-19c575e0cb91","Type":"ContainerStarted","Data":"09d522fa0576ba1fc06dd21a1d288b27cce453868d476c00bd8eb6cbe2e8c7e7"}
Oct 31 00:36:24.026096 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:24.026031    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt" podStartSLOduration=1.9059438800000001 podStartE2EDuration="3.026011746s" totalImagesPullingTime="1.120067866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:24.023946294 +0000 UTC m=+1044.245981835" watchObservedRunningTime="2026-10-31 00:36:24.026011746 +0000 UTC m=+1044.248047289"
Oct 31 00:36:24.126869 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:24.126837    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"
Oct 31 00:36:25.349098 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:25.349065    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"]
Oct 31 00:36:25.352657 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:25.352635    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"
Oct 31 00:36:25.367941 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:25.367910    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 31 00:36:25.381236 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:25.381208    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"]
Oct 31 00:36:25.424344 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:25.424309    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cad8e534-8589-407a-bbcf-0bb145b80233-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj\" (UID: \"cad8e534-8589-407a-bbcf-0bb145b80233\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"
Oct 31 00:36:25.424514 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:25.424373    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh675\" (UniqueName: \"kubernetes.io/projected/cad8e534-8589-407a-bbcf-0bb145b80233-kube-api-access-sh675\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj\" (UID: \"cad8e534-8589-407a-bbcf-0bb145b80233\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"
Oct 31 00:36:25.424514 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:25.424481    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cad8e534-8589-407a-bbcf-0bb145b80233-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj\" (UID: \"cad8e534-8589-407a-bbcf-0bb145b80233\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"
Oct 31 00:36:25.424590 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:25.424543    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cad8e534-8589-407a-bbcf-0bb145b80233-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj\" (UID: \"cad8e534-8589-407a-bbcf-0bb145b80233\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"
Oct 31 00:36:25.424590 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:25.424560    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cad8e534-8589-407a-bbcf-0bb145b80233-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj\" (UID: \"cad8e534-8589-407a-bbcf-0bb145b80233\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"
Oct 31 00:36:25.424664 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:25.424591    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cad8e534-8589-407a-bbcf-0bb145b80233-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj\" (UID: \"cad8e534-8589-407a-bbcf-0bb145b80233\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"
Oct 31 00:36:25.525753 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:25.525710    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cad8e534-8589-407a-bbcf-0bb145b80233-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj\" (UID: \"cad8e534-8589-407a-bbcf-0bb145b80233\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"
Oct 31 00:36:25.525922 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:25.525769    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cad8e534-8589-407a-bbcf-0bb145b80233-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj\" (UID: \"cad8e534-8589-407a-bbcf-0bb145b80233\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"
Oct 31 00:36:25.525922 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:25.525819    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh675\" (UniqueName: \"kubernetes.io/projected/cad8e534-8589-407a-bbcf-0bb145b80233-kube-api-access-sh675\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj\" (UID: \"cad8e534-8589-407a-bbcf-0bb145b80233\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"
Oct 31 00:36:25.525922 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:25.525843    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cad8e534-8589-407a-bbcf-0bb145b80233-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj\" (UID: \"cad8e534-8589-407a-bbcf-0bb145b80233\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"
Oct 31 00:36:25.525922 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:25.525873    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cad8e534-8589-407a-bbcf-0bb145b80233-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj\" (UID: \"cad8e534-8589-407a-bbcf-0bb145b80233\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"
Oct 31 00:36:25.525922 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:25.525889    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cad8e534-8589-407a-bbcf-0bb145b80233-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj\" (UID: \"cad8e534-8589-407a-bbcf-0bb145b80233\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"
Oct 31 00:36:25.526286 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:25.526256    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cad8e534-8589-407a-bbcf-0bb145b80233-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj\" (UID: \"cad8e534-8589-407a-bbcf-0bb145b80233\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"
Oct 31 00:36:25.526424 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:25.526321    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cad8e534-8589-407a-bbcf-0bb145b80233-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj\" (UID: \"cad8e534-8589-407a-bbcf-0bb145b80233\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"
Oct 31 00:36:25.526424 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:25.526375    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cad8e534-8589-407a-bbcf-0bb145b80233-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj\" (UID: \"cad8e534-8589-407a-bbcf-0bb145b80233\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"
Oct 31 00:36:25.528047 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:25.528025    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cad8e534-8589-407a-bbcf-0bb145b80233-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj\" (UID: \"cad8e534-8589-407a-bbcf-0bb145b80233\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"
Oct 31 00:36:25.528292 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:25.528276    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cad8e534-8589-407a-bbcf-0bb145b80233-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj\" (UID: \"cad8e534-8589-407a-bbcf-0bb145b80233\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"
Oct 31 00:36:25.549672 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:25.549648    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh675\" (UniqueName: \"kubernetes.io/projected/cad8e534-8589-407a-bbcf-0bb145b80233-kube-api-access-sh675\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj\" (UID: \"cad8e534-8589-407a-bbcf-0bb145b80233\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"
Oct 31 00:36:25.665806 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:25.665771    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"
Oct 31 00:36:25.965166 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:25.965137    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"]
Oct 31 00:36:25.966518 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:36:25.966492    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad8e534_8589_407a_bbcf_0bb145b80233.slice/crio-a6669cdcc7c959cea679078ccb9d428eac3672f39aee84928a664449fa1af760 WatchSource:0}: Error finding container a6669cdcc7c959cea679078ccb9d428eac3672f39aee84928a664449fa1af760: Status 404 returned error can't find the container with id a6669cdcc7c959cea679078ccb9d428eac3672f39aee84928a664449fa1af760
Oct 31 00:36:25.986636 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:25.986598    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj" event={"ID":"cad8e534-8589-407a-bbcf-0bb145b80233","Type":"ContainerStarted","Data":"a6669cdcc7c959cea679078ccb9d428eac3672f39aee84928a664449fa1af760"}
Oct 31 00:36:26.992328 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:26.992287    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj" event={"ID":"cad8e534-8589-407a-bbcf-0bb145b80233","Type":"ContainerStarted","Data":"c2cc939c080215ea66684d6ddd0a0a9497796bca79ebec45e6bc8c7c4546978f"}
Oct 31 00:36:27.038562 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:27.038509    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj" podStartSLOduration=1.859466208 podStartE2EDuration="2.038496366s" totalImagesPullingTime="179.030158ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:27.036477859 +0000 UTC m=+1047.258513401" watchObservedRunningTime="2026-10-31 00:36:27.038496366 +0000 UTC m=+1047.260531908"
Oct 31 00:36:27.666690 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:27.666651    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"
Oct 31 00:36:29.984980 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:29.984943    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj"
Oct 31 00:36:29.997457 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:29.997427    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj"
Oct 31 00:36:30.016230 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:30.016205    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj"
Oct 31 00:36:31.547631 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:31.547600    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"]
Oct 31 00:36:31.551469 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:31.551444    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"
Oct 31 00:36:31.573933 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:31.573902    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"]
Oct 31 00:36:31.576903 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:31.576876    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v557\" (UniqueName: \"kubernetes.io/projected/e2f2f0dd-a9a0-4335-9c33-683562fa8fbe-kube-api-access-4v557\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s24sn\" (UID: \"e2f2f0dd-a9a0-4335-9c33-683562fa8fbe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"
Oct 31 00:36:31.577016 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:31.576927    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2f2f0dd-a9a0-4335-9c33-683562fa8fbe-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s24sn\" (UID: \"e2f2f0dd-a9a0-4335-9c33-683562fa8fbe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"
Oct 31 00:36:31.577056 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:31.577028    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2f2f0dd-a9a0-4335-9c33-683562fa8fbe-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s24sn\" (UID: \"e2f2f0dd-a9a0-4335-9c33-683562fa8fbe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"
Oct 31 00:36:31.577108 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:31.577084    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2f2f0dd-a9a0-4335-9c33-683562fa8fbe-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s24sn\" (UID: \"e2f2f0dd-a9a0-4335-9c33-683562fa8fbe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"
Oct 31 00:36:31.577150 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:31.577130    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2f2f0dd-a9a0-4335-9c33-683562fa8fbe-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s24sn\" (UID: \"e2f2f0dd-a9a0-4335-9c33-683562fa8fbe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"
Oct 31 00:36:31.577219 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:31.577203    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2f2f0dd-a9a0-4335-9c33-683562fa8fbe-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s24sn\" (UID: \"e2f2f0dd-a9a0-4335-9c33-683562fa8fbe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"
Oct 31 00:36:31.581381 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:31.581362    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 31 00:36:31.677866 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:31.677827    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2f2f0dd-a9a0-4335-9c33-683562fa8fbe-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s24sn\" (UID: \"e2f2f0dd-a9a0-4335-9c33-683562fa8fbe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"
Oct 31 00:36:31.678061 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:31.677876    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2f2f0dd-a9a0-4335-9c33-683562fa8fbe-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s24sn\" (UID: \"e2f2f0dd-a9a0-4335-9c33-683562fa8fbe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"
Oct 31 00:36:31.678061 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:31.677908    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2f2f0dd-a9a0-4335-9c33-683562fa8fbe-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s24sn\" (UID: \"e2f2f0dd-a9a0-4335-9c33-683562fa8fbe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"
Oct 31 00:36:31.678061 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:31.677927    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2f2f0dd-a9a0-4335-9c33-683562fa8fbe-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s24sn\" (UID: \"e2f2f0dd-a9a0-4335-9c33-683562fa8fbe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"
Oct 31 00:36:31.678061 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:31.677961    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2f2f0dd-a9a0-4335-9c33-683562fa8fbe-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s24sn\" (UID: \"e2f2f0dd-a9a0-4335-9c33-683562fa8fbe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"
Oct 31 00:36:31.678061 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:31.678007    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v557\" (UniqueName: \"kubernetes.io/projected/e2f2f0dd-a9a0-4335-9c33-683562fa8fbe-kube-api-access-4v557\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s24sn\" (UID: \"e2f2f0dd-a9a0-4335-9c33-683562fa8fbe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"
Oct 31 00:36:31.678418 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:31.678383    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2f2f0dd-a9a0-4335-9c33-683562fa8fbe-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s24sn\" (UID: \"e2f2f0dd-a9a0-4335-9c33-683562fa8fbe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"
Oct 31 00:36:31.678418 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:31.678411    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2f2f0dd-a9a0-4335-9c33-683562fa8fbe-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s24sn\" (UID: \"e2f2f0dd-a9a0-4335-9c33-683562fa8fbe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"
Oct 31 00:36:31.678558 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:31.678447    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2f2f0dd-a9a0-4335-9c33-683562fa8fbe-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s24sn\" (UID: \"e2f2f0dd-a9a0-4335-9c33-683562fa8fbe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"
Oct 31 00:36:31.680101 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:31.680085    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2f2f0dd-a9a0-4335-9c33-683562fa8fbe-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s24sn\" (UID: \"e2f2f0dd-a9a0-4335-9c33-683562fa8fbe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"
Oct 31 00:36:31.680445 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:31.680426    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2f2f0dd-a9a0-4335-9c33-683562fa8fbe-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s24sn\" (UID: \"e2f2f0dd-a9a0-4335-9c33-683562fa8fbe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"
Oct 31 00:36:31.705282 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:31.705247    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v557\" (UniqueName: \"kubernetes.io/projected/e2f2f0dd-a9a0-4335-9c33-683562fa8fbe-kube-api-access-4v557\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s24sn\" (UID: \"e2f2f0dd-a9a0-4335-9c33-683562fa8fbe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"
Oct 31 00:36:31.868769 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:31.868728    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"
Oct 31 00:36:32.126795 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:32.126726    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"
Oct 31 00:36:32.139364 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:32.139337    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"
Oct 31 00:36:32.166301 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:36:32.166271    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f2f0dd_a9a0_4335_9c33_683562fa8fbe.slice/crio-fd295e4e3a8474f00cc7db772c302d0b37d8665e59247fb9103aa56d00083a5c WatchSource:0}: Error finding container fd295e4e3a8474f00cc7db772c302d0b37d8665e59247fb9103aa56d00083a5c: Status 404 returned error can't find the container with id fd295e4e3a8474f00cc7db772c302d0b37d8665e59247fb9103aa56d00083a5c
Oct 31 00:36:32.170045 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:32.170021    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"]
Oct 31 00:36:33.018406 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.018361    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn" event={"ID":"e2f2f0dd-a9a0-4335-9c33-683562fa8fbe","Type":"ContainerStarted","Data":"121a17b2941b2d8a25cb0a242fa3003493956ee1fd2c9b9d704c28b4db375fbf"}
Oct 31 00:36:33.018840 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.018426    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn" event={"ID":"e2f2f0dd-a9a0-4335-9c33-683562fa8fbe","Type":"ContainerStarted","Data":"fd295e4e3a8474f00cc7db772c302d0b37d8665e59247fb9103aa56d00083a5c"}
Oct 31 00:36:33.030221 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.030194    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt"
Oct 31 00:36:33.063236 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.063187    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn" podStartSLOduration=1.865996663 podStartE2EDuration="2.063170863s" totalImagesPullingTime="197.1742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:33.061886597 +0000 UTC m=+1053.283922159" watchObservedRunningTime="2026-10-31 00:36:33.063170863 +0000 UTC m=+1053.285206404"
Oct 31 00:36:33.706336 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.706304    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"]
Oct 31 00:36:33.709600 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.709580    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"
Oct 31 00:36:33.729344 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.729322    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 31 00:36:33.757546 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.757518    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"]
Oct 31 00:36:33.797415 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.797386    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c1e66f4-2552-43b1-96e0-272661898154-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q\" (UID: \"7c1e66f4-2552-43b1-96e0-272661898154\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"
Oct 31 00:36:33.797415 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.797419    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c1e66f4-2552-43b1-96e0-272661898154-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q\" (UID: \"7c1e66f4-2552-43b1-96e0-272661898154\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"
Oct 31 00:36:33.797614 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.797443    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c1e66f4-2552-43b1-96e0-272661898154-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q\" (UID: \"7c1e66f4-2552-43b1-96e0-272661898154\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"
Oct 31 00:36:33.797614 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.797569    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c1e66f4-2552-43b1-96e0-272661898154-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q\" (UID: \"7c1e66f4-2552-43b1-96e0-272661898154\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"
Oct 31 00:36:33.797614 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.797606    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c1e66f4-2552-43b1-96e0-272661898154-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q\" (UID: \"7c1e66f4-2552-43b1-96e0-272661898154\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"
Oct 31 00:36:33.797725 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.797690    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbk4c\" (UniqueName: \"kubernetes.io/projected/7c1e66f4-2552-43b1-96e0-272661898154-kube-api-access-mbk4c\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q\" (UID: \"7c1e66f4-2552-43b1-96e0-272661898154\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"
Oct 31 00:36:33.869887 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.869851    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"
Oct 31 00:36:33.898419 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.898389    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c1e66f4-2552-43b1-96e0-272661898154-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q\" (UID: \"7c1e66f4-2552-43b1-96e0-272661898154\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"
Oct 31 00:36:33.898419 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.898419    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c1e66f4-2552-43b1-96e0-272661898154-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q\" (UID: \"7c1e66f4-2552-43b1-96e0-272661898154\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"
Oct 31 00:36:33.898607 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.898439    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c1e66f4-2552-43b1-96e0-272661898154-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q\" (UID: \"7c1e66f4-2552-43b1-96e0-272661898154\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"
Oct 31 00:36:33.898607 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.898469    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c1e66f4-2552-43b1-96e0-272661898154-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q\" (UID: \"7c1e66f4-2552-43b1-96e0-272661898154\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"
Oct 31 00:36:33.898607 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.898497    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c1e66f4-2552-43b1-96e0-272661898154-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q\" (UID: \"7c1e66f4-2552-43b1-96e0-272661898154\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"
Oct 31 00:36:33.898607 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.898547    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbk4c\" (UniqueName: \"kubernetes.io/projected/7c1e66f4-2552-43b1-96e0-272661898154-kube-api-access-mbk4c\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q\" (UID: \"7c1e66f4-2552-43b1-96e0-272661898154\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"
Oct 31 00:36:33.898941 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.898920    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c1e66f4-2552-43b1-96e0-272661898154-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q\" (UID: \"7c1e66f4-2552-43b1-96e0-272661898154\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"
Oct 31 00:36:33.899036 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.898991    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c1e66f4-2552-43b1-96e0-272661898154-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q\" (UID: \"7c1e66f4-2552-43b1-96e0-272661898154\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"
Oct 31 00:36:33.899081 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.899070    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c1e66f4-2552-43b1-96e0-272661898154-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q\" (UID: \"7c1e66f4-2552-43b1-96e0-272661898154\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"
Oct 31 00:36:33.900843 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.900819    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c1e66f4-2552-43b1-96e0-272661898154-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q\" (UID: \"7c1e66f4-2552-43b1-96e0-272661898154\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"
Oct 31 00:36:33.901038 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.901021    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c1e66f4-2552-43b1-96e0-272661898154-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q\" (UID: \"7c1e66f4-2552-43b1-96e0-272661898154\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"
Oct 31 00:36:33.926227 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:33.926196    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbk4c\" (UniqueName: \"kubernetes.io/projected/7c1e66f4-2552-43b1-96e0-272661898154-kube-api-access-mbk4c\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q\" (UID: \"7c1e66f4-2552-43b1-96e0-272661898154\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"
Oct 31 00:36:34.022461 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:34.022376    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"
Oct 31 00:36:34.317705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:34.317665    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"]
Oct 31 00:36:34.318353 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:36:34.318331    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1e66f4_2552_43b1_96e0_272661898154.slice/crio-df860a1499d7bd28f8ec17ab97c46c00e744fa1b47a632add14ca24cc9d91a6e WatchSource:0}: Error finding container df860a1499d7bd28f8ec17ab97c46c00e744fa1b47a632add14ca24cc9d91a6e: Status 404 returned error can't find the container with id df860a1499d7bd28f8ec17ab97c46c00e744fa1b47a632add14ca24cc9d91a6e
Oct 31 00:36:35.028367 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:35.028333    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q" event={"ID":"7c1e66f4-2552-43b1-96e0-272661898154","Type":"ContainerStarted","Data":"ee05c9c677186a67ae0f4d6396487b478b9dca236c236867dbf42fc9823ac888"}
Oct 31 00:36:35.028881 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:35.028394    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q" event={"ID":"7c1e66f4-2552-43b1-96e0-272661898154","Type":"ContainerStarted","Data":"df860a1499d7bd28f8ec17ab97c46c00e744fa1b47a632add14ca24cc9d91a6e"}
Oct 31 00:36:35.072860 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:35.072804    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q" podStartSLOduration=1.903745121 podStartE2EDuration="2.072789937s" totalImagesPullingTime="169.044816ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:35.072119629 +0000 UTC m=+1055.294155191" watchObservedRunningTime="2026-10-31 00:36:35.072789937 +0000 UTC m=+1055.294825483"
Oct 31 00:36:35.666129 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:35.666091    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"
Oct 31 00:36:35.678584 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:35.678554    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"
Oct 31 00:36:36.022623 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:36.022544    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"
Oct 31 00:36:36.045162 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:36.045127    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj"
Oct 31 00:36:41.868922 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:41.868885    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"
Oct 31 00:36:41.882203 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:41.882172    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"
Oct 31 00:36:42.072799 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:42.072771    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s24sn"
Oct 31 00:36:44.022881 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:44.022850    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"
Oct 31 00:36:44.036017 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:44.035984    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"
Oct 31 00:36:44.082383 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:44.082353    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q"
Oct 31 00:36:45.291338 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:36:45.291301    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:36:51.022337 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:51.022299    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-74d4469f5d-6vq6q"]
Oct 31 00:36:51.025543 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:51.025521    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-74d4469f5d-6vq6q"
Oct 31 00:36:51.042945 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:51.042922    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 31 00:36:51.055564 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:51.055534    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 31 00:36:51.055725 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:51.055627    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-mtk2r\""
Oct 31 00:36:51.057076 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:51.057057    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-74d4469f5d-6vq6q"]
Oct 31 00:36:51.149320 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:51.149280    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e-metrics-certs\") pod \"ai-gateway-controller-74d4469f5d-6vq6q\" (UID: \"f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e\") " pod="opendatahub/ai-gateway-controller-74d4469f5d-6vq6q"
Oct 31 00:36:51.149475 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:51.149330    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m994\" (UniqueName: \"kubernetes.io/projected/f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e-kube-api-access-9m994\") pod \"ai-gateway-controller-74d4469f5d-6vq6q\" (UID: \"f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e\") " pod="opendatahub/ai-gateway-controller-74d4469f5d-6vq6q"
Oct 31 00:36:51.249862 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:51.249830    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e-metrics-certs\") pod \"ai-gateway-controller-74d4469f5d-6vq6q\" (UID: \"f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e\") " pod="opendatahub/ai-gateway-controller-74d4469f5d-6vq6q"
Oct 31 00:36:51.249862 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:51.249866    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m994\" (UniqueName: \"kubernetes.io/projected/f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e-kube-api-access-9m994\") pod \"ai-gateway-controller-74d4469f5d-6vq6q\" (UID: \"f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e\") " pod="opendatahub/ai-gateway-controller-74d4469f5d-6vq6q"
Oct 31 00:36:51.252247 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:51.252225    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e-metrics-certs\") pod \"ai-gateway-controller-74d4469f5d-6vq6q\" (UID: \"f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e\") " pod="opendatahub/ai-gateway-controller-74d4469f5d-6vq6q"
Oct 31 00:36:51.273729 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:51.273658    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m994\" (UniqueName: \"kubernetes.io/projected/f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e-kube-api-access-9m994\") pod \"ai-gateway-controller-74d4469f5d-6vq6q\" (UID: \"f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e\") " pod="opendatahub/ai-gateway-controller-74d4469f5d-6vq6q"
Oct 31 00:36:51.338729 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:51.338675    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-74d4469f5d-6vq6q"
Oct 31 00:36:51.631329 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:51.631304    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-74d4469f5d-6vq6q"]
Oct 31 00:36:51.632276 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:36:51.632250    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e3b76d_bfd2_4eaf_a6f8_f36b90a6919e.slice/crio-6e3663c986e295d2f1da6ec1d7ebf026bf5c8c39e2a26242ca98d38008324863 WatchSource:0}: Error finding container 6e3663c986e295d2f1da6ec1d7ebf026bf5c8c39e2a26242ca98d38008324863: Status 404 returned error can't find the container with id 6e3663c986e295d2f1da6ec1d7ebf026bf5c8c39e2a26242ca98d38008324863
Oct 31 00:36:52.103914 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:52.103874    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-74d4469f5d-6vq6q" event={"ID":"f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e","Type":"ContainerStarted","Data":"6e3663c986e295d2f1da6ec1d7ebf026bf5c8c39e2a26242ca98d38008324863"}
Oct 31 00:36:54.115410 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:54.115374    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-74d4469f5d-6vq6q" event={"ID":"f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e","Type":"ContainerStarted","Data":"a9f3189289245033a953ff973f41a65e536c841652902d0307168aa143e1b5f3"}
Oct 31 00:36:54.115905 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:54.115488    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-74d4469f5d-6vq6q"
Oct 31 00:36:54.157896 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:54.157844    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-74d4469f5d-6vq6q" podStartSLOduration=2.703144309 podStartE2EDuration="4.157831948s" totalImagesPullingTime="1.454687639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:54.156589979 +0000 UTC m=+1074.378625530" watchObservedRunningTime="2026-10-31 00:36:54.157831948 +0000 UTC m=+1074.379867492"
Oct 31 00:36:55.490974 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:55.490941    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-64nxw"]
Oct 31 00:36:55.494585 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:55.494568    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-64nxw"
Oct 31 00:36:55.516152 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:55.516124    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-64nxw"]
Oct 31 00:36:55.552332 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:55.552298    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 31 00:36:55.553300 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:55.553271    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-dtl5x\""
Oct 31 00:36:55.589772 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:55.589740    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6mwf\" (UniqueName: \"kubernetes.io/projected/852f9567-a315-448d-b286-f90afbcaad43-kube-api-access-t6mwf\") pod \"payload-pre-processing-7b98bb678c-64nxw\" (UID: \"852f9567-a315-448d-b286-f90afbcaad43\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-64nxw"
Oct 31 00:36:55.589906 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:55.589809    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/852f9567-a315-448d-b286-f90afbcaad43-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-64nxw\" (UID: \"852f9567-a315-448d-b286-f90afbcaad43\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-64nxw"
Oct 31 00:36:55.691343 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:55.691310    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6mwf\" (UniqueName: \"kubernetes.io/projected/852f9567-a315-448d-b286-f90afbcaad43-kube-api-access-t6mwf\") pod \"payload-pre-processing-7b98bb678c-64nxw\" (UID: \"852f9567-a315-448d-b286-f90afbcaad43\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-64nxw"
Oct 31 00:36:55.691522 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:55.691364    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/852f9567-a315-448d-b286-f90afbcaad43-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-64nxw\" (UID: \"852f9567-a315-448d-b286-f90afbcaad43\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-64nxw"
Oct 31 00:36:55.692111 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:55.692072    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/852f9567-a315-448d-b286-f90afbcaad43-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-64nxw\" (UID: \"852f9567-a315-448d-b286-f90afbcaad43\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-64nxw"
Oct 31 00:36:55.714577 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:55.714548    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6mwf\" (UniqueName: \"kubernetes.io/projected/852f9567-a315-448d-b286-f90afbcaad43-kube-api-access-t6mwf\") pod \"payload-pre-processing-7b98bb678c-64nxw\" (UID: \"852f9567-a315-448d-b286-f90afbcaad43\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-64nxw"
Oct 31 00:36:55.806673 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:55.806581    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-64nxw"
Oct 31 00:36:56.103103 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:56.103069    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-64nxw"]
Oct 31 00:36:56.103558 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:36:56.103531    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852f9567_a315_448d_b286_f90afbcaad43.slice/crio-89cac1e220e848548c80febdb2131c5ef3bc1bc3c4e608375663ab7e740ccf04 WatchSource:0}: Error finding container 89cac1e220e848548c80febdb2131c5ef3bc1bc3c4e608375663ab7e740ccf04: Status 404 returned error can't find the container with id 89cac1e220e848548c80febdb2131c5ef3bc1bc3c4e608375663ab7e740ccf04
Oct 31 00:36:56.125112 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:56.125079    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-64nxw" event={"ID":"852f9567-a315-448d-b286-f90afbcaad43","Type":"ContainerStarted","Data":"89cac1e220e848548c80febdb2131c5ef3bc1bc3c4e608375663ab7e740ccf04"}
Oct 31 00:36:56.835270 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:56.835235    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"]
Oct 31 00:36:56.838691 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:56.838662    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:56.853766 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:56.853743    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-vtxt8\""
Oct 31 00:36:56.868211 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:56.868179    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"]
Oct 31 00:36:56.900603 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:56.900569    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:56.901047 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:56.900618    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:56.901047 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:56.900759    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:56.901047 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:56.900812    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:56.901047 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:56.900845    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:56.901047 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:56.900892    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:56.901047 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:56.900957    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:56.901047 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:56.900979    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:56.901047 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:56.901026    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfdfk\" (UniqueName: \"kubernetes.io/projected/820c95b2-cfd6-4893-80ed-1c746d4cfef2-kube-api-access-nfdfk\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:57.002566 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:57.002527    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfdfk\" (UniqueName: \"kubernetes.io/projected/820c95b2-cfd6-4893-80ed-1c746d4cfef2-kube-api-access-nfdfk\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:57.002756 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:57.002587    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:57.002756 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:57.002618    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:57.002875 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:57.002792    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:57.002875 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:57.002848    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:57.003000 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:57.002882    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:57.003000 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:57.002935    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:57.003103 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:57.003035    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:57.003103 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:57.003088    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:57.003387 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:57.003358    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:57.003519 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:57.003474    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:57.003752 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:57.003730    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:57.004023 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:57.003999    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:57.004192 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:57.004169    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:57.006067 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:57.006044    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:57.006205 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:57.006112    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:57.026456 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:57.026420    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:57.026591 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:57.026539    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfdfk\" (UniqueName: \"kubernetes.io/projected/820c95b2-cfd6-4893-80ed-1c746d4cfef2-kube-api-access-nfdfk\") pod \"maas-default-gateway-openshift-default-6886b96f9-8zxnz\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:57.154781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:57.154745    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:36:57.477330 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:57.477297    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"]
Oct 31 00:36:57.478175 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:36:57.478142    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod820c95b2_cfd6_4893_80ed_1c746d4cfef2.slice/crio-092482c52236f71627a23b8199cb99e70a37456af0a6796222364e9f44f23d99 WatchSource:0}: Error finding container 092482c52236f71627a23b8199cb99e70a37456af0a6796222364e9f44f23d99: Status 404 returned error can't find the container with id 092482c52236f71627a23b8199cb99e70a37456af0a6796222364e9f44f23d99
Oct 31 00:36:58.142764 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:58.142721    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz" event={"ID":"820c95b2-cfd6-4893-80ed-1c746d4cfef2","Type":"ContainerStarted","Data":"092482c52236f71627a23b8199cb99e70a37456af0a6796222364e9f44f23d99"}
Oct 31 00:36:59.149583 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:59.149535    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-64nxw" event={"ID":"852f9567-a315-448d-b286-f90afbcaad43","Type":"ContainerStarted","Data":"46863398f1ab7cbd076277b61bbbc9fcc08001178a21b7499df504d639838363"}
Oct 31 00:36:59.150018 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:59.149674    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-64nxw"
Oct 31 00:36:59.191248 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:36:59.191185    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-64nxw" podStartSLOduration=2.113422938 podStartE2EDuration="4.191166711s" totalImagesPullingTime="2.077743773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:59.190235409 +0000 UTC m=+1079.412270951" watchObservedRunningTime="2026-10-31 00:36:59.191166711 +0000 UTC m=+1079.413202254"
Oct 31 00:37:00.991298 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:00.991258    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:37:00.991719 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:00.991345    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:37:00.991719 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:00.991394    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:37:01.161216 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:01.161170    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz" event={"ID":"820c95b2-cfd6-4893-80ed-1c746d4cfef2","Type":"ContainerStarted","Data":"a1711b8c06eb31b128c553f8ccd5bd0a61636ff87d8c23bfe20688aaedc373a4"}
Oct 31 00:37:01.213975 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:01.213919    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz" podStartSLOduration=1.704243381 podStartE2EDuration="5.213905685s" totalImagesPullingTime="3.509662304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:01.211598094 +0000 UTC m=+1081.433633678" watchObservedRunningTime="2026-10-31 00:37:01.213905685 +0000 UTC m=+1081.435941226"
Oct 31 00:37:02.155566 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:02.155533    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:37:02.156996 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:02.156972    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8zxnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 31 00:37:02.157058 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:02.157021    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz" podUID="820c95b2-cfd6-4893-80ed-1c746d4cfef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 31 00:37:03.155117 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:03.155080    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8zxnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 31 00:37:03.155294 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:03.155139    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz" podUID="820c95b2-cfd6-4893-80ed-1c746d4cfef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 31 00:37:04.155797 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:04.155760    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8zxnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 31 00:37:04.156201 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:04.155823    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz" podUID="820c95b2-cfd6-4893-80ed-1c746d4cfef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 31 00:37:05.137969 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:05.137932    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-74d4469f5d-6vq6q"
Oct 31 00:37:05.156104 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:05.156074    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8zxnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 31 00:37:05.156502 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:05.156127    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz" podUID="820c95b2-cfd6-4893-80ed-1c746d4cfef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 31 00:37:05.413487 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:05.413404    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-79889fd95f-btjgl"]
Oct 31 00:37:05.438023 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:05.437993    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-79889fd95f-btjgl"
Oct 31 00:37:05.445484 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:05.445452    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-79889fd95f-btjgl"]
Oct 31 00:37:05.588184 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:05.588143    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj25r\" (UniqueName: \"kubernetes.io/projected/5cfa01dd-f048-4498-ba79-629852aaff37-kube-api-access-pj25r\") pod \"ai-gateway-controller-79889fd95f-btjgl\" (UID: \"5cfa01dd-f048-4498-ba79-629852aaff37\") " pod="opendatahub/ai-gateway-controller-79889fd95f-btjgl"
Oct 31 00:37:05.588383 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:05.588211    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cfa01dd-f048-4498-ba79-629852aaff37-metrics-certs\") pod \"ai-gateway-controller-79889fd95f-btjgl\" (UID: \"5cfa01dd-f048-4498-ba79-629852aaff37\") " pod="opendatahub/ai-gateway-controller-79889fd95f-btjgl"
Oct 31 00:37:05.688852 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:05.688756    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj25r\" (UniqueName: \"kubernetes.io/projected/5cfa01dd-f048-4498-ba79-629852aaff37-kube-api-access-pj25r\") pod \"ai-gateway-controller-79889fd95f-btjgl\" (UID: \"5cfa01dd-f048-4498-ba79-629852aaff37\") " pod="opendatahub/ai-gateway-controller-79889fd95f-btjgl"
Oct 31 00:37:05.688852 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:05.688821    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cfa01dd-f048-4498-ba79-629852aaff37-metrics-certs\") pod \"ai-gateway-controller-79889fd95f-btjgl\" (UID: \"5cfa01dd-f048-4498-ba79-629852aaff37\") " pod="opendatahub/ai-gateway-controller-79889fd95f-btjgl"
Oct 31 00:37:05.691208 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:05.691188    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cfa01dd-f048-4498-ba79-629852aaff37-metrics-certs\") pod \"ai-gateway-controller-79889fd95f-btjgl\" (UID: \"5cfa01dd-f048-4498-ba79-629852aaff37\") " pod="opendatahub/ai-gateway-controller-79889fd95f-btjgl"
Oct 31 00:37:05.712670 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:05.712641    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj25r\" (UniqueName: \"kubernetes.io/projected/5cfa01dd-f048-4498-ba79-629852aaff37-kube-api-access-pj25r\") pod \"ai-gateway-controller-79889fd95f-btjgl\" (UID: \"5cfa01dd-f048-4498-ba79-629852aaff37\") " pod="opendatahub/ai-gateway-controller-79889fd95f-btjgl"
Oct 31 00:37:05.755596 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:05.755557    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-79889fd95f-btjgl"
Oct 31 00:37:06.052387 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:06.052351    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-79889fd95f-btjgl"]
Oct 31 00:37:06.053667 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:37:06.053641    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cfa01dd_f048_4498_ba79_629852aaff37.slice/crio-707397e42dbe49bb351532347a10326314047866a79985b17c15d83b66799710 WatchSource:0}: Error finding container 707397e42dbe49bb351532347a10326314047866a79985b17c15d83b66799710: Status 404 returned error can't find the container with id 707397e42dbe49bb351532347a10326314047866a79985b17c15d83b66799710
Oct 31 00:37:06.155243 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:06.155205    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8zxnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 31 00:37:06.155419 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:06.155261    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz" podUID="820c95b2-cfd6-4893-80ed-1c746d4cfef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 31 00:37:06.189520 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:06.189482    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-79889fd95f-btjgl" event={"ID":"5cfa01dd-f048-4498-ba79-629852aaff37","Type":"ContainerStarted","Data":"707397e42dbe49bb351532347a10326314047866a79985b17c15d83b66799710"}
Oct 31 00:37:07.155456 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:07.155420    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8zxnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 31 00:37:07.155632 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:07.155474    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz" podUID="820c95b2-cfd6-4893-80ed-1c746d4cfef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 31 00:37:07.195711 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:07.195658    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-79889fd95f-btjgl" event={"ID":"5cfa01dd-f048-4498-ba79-629852aaff37","Type":"ContainerStarted","Data":"8ab24596f9e946592626dd3a6965ffda1f423984fab8021854999b5c189255c9"}
Oct 31 00:37:07.196199 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:07.195737    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-79889fd95f-btjgl"
Oct 31 00:37:07.242670 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:07.242595    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-79889fd95f-btjgl" podStartSLOduration=1.92237691 podStartE2EDuration="2.242576114s" totalImagesPullingTime="320.199204ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:07.240518592 +0000 UTC m=+1087.462554146" watchObservedRunningTime="2026-10-31 00:37:07.242576114 +0000 UTC m=+1087.464611657"
Oct 31 00:37:08.155805 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:08.155770    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8zxnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 31 00:37:08.155990 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:08.155822    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz" podUID="820c95b2-cfd6-4893-80ed-1c746d4cfef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 31 00:37:09.243478 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:09.243440    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:37:09.243938 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:09.243878    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:37:09.244714 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:09.244692    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 00:37:10.157971 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:10.157942    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-64nxw"
Oct 31 00:37:11.842177 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:11.842140    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"]
Oct 31 00:37:11.846076 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:11.846045    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:11.878195 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:11.878156    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"]
Oct 31 00:37:11.883251 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:11.883222    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"]
Oct 31 00:37:11.886925 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:11.886905    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:11.921365 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:11.921335    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"]
Oct 31 00:37:11.943075 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:11.943046    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:11.943075 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:11.943082    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:11.943290 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:11.943102    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:11.943290 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:11.943249    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:11.943373 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:11.943288    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:11.943373 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:11.943345    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:11.943445 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:11.943370    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:11.943445 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:11.943419    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:11.943514 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:11.943441    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s52gp\" (UniqueName: \"kubernetes.io/projected/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-kube-api-access-s52gp\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:11.943514 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:11.943500    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:11.943588 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:11.943538    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:11.943622 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:11.943574    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f7f289-ef35-494d-91c9-034c7f1294d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:11.943663 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:11.943644    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:11.943721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:11.943660    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:11.943721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:11.943715    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f877v\" (UniqueName: \"kubernetes.io/projected/a5f7f289-ef35-494d-91c9-034c7f1294d4-kube-api-access-f877v\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:11.943790 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:11.943735    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:11.943790 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:11.943758    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:11.943790 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:11.943776    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:12.045717 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.045668    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:12.045912 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.045755    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:12.045912 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.045775    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:12.045912 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.045793    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:12.045912 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.045815    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:12.045912 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.045873    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:12.046154 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.045914    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s52gp\" (UniqueName: \"kubernetes.io/projected/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-kube-api-access-s52gp\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:12.046154 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.045945    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:12.046154 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.046091    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:12.046154 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.046124    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f7f289-ef35-494d-91c9-034c7f1294d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:12.046361 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.046166    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:12.046361 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.046192    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:12.046361 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.046216    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f877v\" (UniqueName: \"kubernetes.io/projected/a5f7f289-ef35-494d-91c9-034c7f1294d4-kube-api-access-f877v\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:12.046361 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.046245    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:12.046361 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.046248    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:12.046361 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.046283    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:12.046361 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.046320    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:12.046722 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.046373    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:12.046722 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.046387    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:12.046722 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.046417    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:12.046893 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.046733    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:12.046893 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.046800    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:12.046893 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.046868    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f7f289-ef35-494d-91c9-034c7f1294d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:12.047062 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.046905    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:12.047062 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.046926    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:12.047180 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.047072    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:12.047234 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.047178    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:12.047293 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.047257    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:12.048598 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.048576    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:12.048942 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.048922    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:12.049154 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.049137    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:12.049446 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.049430    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:12.069875 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.069844    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:12.070086 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.070064    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:12.070559 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.070536    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s52gp\" (UniqueName: \"kubernetes.io/projected/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-kube-api-access-s52gp\") pod \"maas-default-gateway-openshift-default-6886b96f9-4qhrd\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:12.070645 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.070592    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f877v\" (UniqueName: \"kubernetes.io/projected/a5f7f289-ef35-494d-91c9-034c7f1294d4-kube-api-access-f877v\") pod \"maas-default-gateway-openshift-default-6886b96f9-hk9qf\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:12.158823 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.158785    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:12.204323 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.204279    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:12.473634 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.473597    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"]
Oct 31 00:37:12.473958 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:37:12.473936    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f7f289_ef35_494d_91c9_034c7f1294d4.slice/crio-0382969f169860e646ea6ce5ab71e97658d7d4a8cf3d2f6668684e9c6adf6bc8 WatchSource:0}: Error finding container 0382969f169860e646ea6ce5ab71e97658d7d4a8cf3d2f6668684e9c6adf6bc8: Status 404 returned error can't find the container with id 0382969f169860e646ea6ce5ab71e97658d7d4a8cf3d2f6668684e9c6adf6bc8
Oct 31 00:37:12.476985 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.476954    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:37:12.477056 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.477022    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:37:12.477056 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.477051    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:37:12.520172 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.520146    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"]
Oct 31 00:37:12.520377 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:37:12.520356    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af61687_4a18_4cf5_85e0_aa2ae8a2a2d3.slice/crio-204b13b58dd9dd1620f2cc5497bd95645eb292ed57d9a517fd628440ee5510ce WatchSource:0}: Error finding container 204b13b58dd9dd1620f2cc5497bd95645eb292ed57d9a517fd628440ee5510ce: Status 404 returned error can't find the container with id 204b13b58dd9dd1620f2cc5497bd95645eb292ed57d9a517fd628440ee5510ce
Oct 31 00:37:12.523469 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.523437    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:37:12.523546 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.523500    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:37:12.523546 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:12.523529    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:37:13.269626 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:13.269582    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd" event={"ID":"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3","Type":"ContainerStarted","Data":"b3268d824f6459a5ce45f602b372e77676b9abc1eee0ed4dc6232c980ca0d3bf"}
Oct 31 00:37:13.270121 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:13.270084    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd" event={"ID":"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3","Type":"ContainerStarted","Data":"204b13b58dd9dd1620f2cc5497bd95645eb292ed57d9a517fd628440ee5510ce"}
Oct 31 00:37:13.271652 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:13.271627    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf" event={"ID":"a5f7f289-ef35-494d-91c9-034c7f1294d4","Type":"ContainerStarted","Data":"d7c679a4f978fe103be5bb8f73307dc8a715d9159088dcc7e6c4c82e148cfa71"}
Oct 31 00:37:13.271764 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:13.271698    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf" event={"ID":"a5f7f289-ef35-494d-91c9-034c7f1294d4","Type":"ContainerStarted","Data":"0382969f169860e646ea6ce5ab71e97658d7d4a8cf3d2f6668684e9c6adf6bc8"}
Oct 31 00:37:13.344702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:13.344355    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd" podStartSLOduration=2.344333312 podStartE2EDuration="2.344333312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:13.343021792 +0000 UTC m=+1093.565057352" watchObservedRunningTime="2026-10-31 00:37:13.344333312 +0000 UTC m=+1093.566368855"
Oct 31 00:37:13.397957 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:13.397899    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf" podStartSLOduration=2.397885355 podStartE2EDuration="2.397885355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:13.39707887 +0000 UTC m=+1093.619114430" watchObservedRunningTime="2026-10-31 00:37:13.397885355 +0000 UTC m=+1093.619920897"
Oct 31 00:37:14.161641 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:14.161051    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:14.163303 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:14.162944    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hk9qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 31 00:37:14.163303 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:14.163000    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf" podUID="a5f7f289-ef35-494d-91c9-034c7f1294d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 31 00:37:14.205059 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:14.204986    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:14.206122 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:14.206103    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4qhrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 31 00:37:14.206276 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:14.206258    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd" podUID="6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 31 00:37:15.161622 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:15.161127    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hk9qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 31 00:37:15.161622 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:15.161207    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf" podUID="a5f7f289-ef35-494d-91c9-034c7f1294d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 31 00:37:15.204994 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:15.204957    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4qhrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 31 00:37:15.205165 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:15.205016    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd" podUID="6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 31 00:37:15.292059 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:37:15.292021    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:37:16.161702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:16.161134    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hk9qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 31 00:37:16.161702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:16.161202    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf" podUID="a5f7f289-ef35-494d-91c9-034c7f1294d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 31 00:37:16.204865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:16.204828    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4qhrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 31 00:37:16.205035 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:16.204884    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd" podUID="6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 31 00:37:17.163702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:17.162213    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hk9qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 31 00:37:17.163702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:17.162294    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf" podUID="a5f7f289-ef35-494d-91c9-034c7f1294d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 31 00:37:17.204758 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:17.204716    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4qhrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 31 00:37:17.204932 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:17.204771    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd" podUID="6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 31 00:37:18.161704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:18.161097    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hk9qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 31 00:37:18.161704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:18.161160    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf" podUID="a5f7f289-ef35-494d-91c9-034c7f1294d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 31 00:37:18.203302 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:18.203273    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-79889fd95f-btjgl"
Oct 31 00:37:18.204857 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:18.204836    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4qhrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 31 00:37:18.204946 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:18.204880    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd" podUID="6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 31 00:37:18.283726 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:18.283653    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-74d4469f5d-6vq6q"]
Oct 31 00:37:18.283997 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:18.283966    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-74d4469f5d-6vq6q" podUID="f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e" containerName="manager" containerID="cri-o://a9f3189289245033a953ff973f41a65e536c841652902d0307168aa143e1b5f3" gracePeriod=30
Oct 31 00:37:18.598367 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:18.598333    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-74d4469f5d-6vq6q"
Oct 31 00:37:18.622005 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:18.621964    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e-metrics-certs\") pod \"f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e\" (UID: \"f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e\") "
Oct 31 00:37:18.622183 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:18.622029    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e-kube-api-access-9m994\" (UniqueName: \"kubernetes.io/projected/f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e-kube-api-access-9m994\") pod \"f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e\" (UID: \"f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e\") "
Oct 31 00:37:18.624019 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:18.623987    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e-metrics-certs" pod "f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e" (UID: "f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:37:18.624159 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:18.624142    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e-kube-api-access-9m994" pod "f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e" (UID: "f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e"). InnerVolumeSpecName "kube-api-access-9m994". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:37:18.723273 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:18.723233    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:37:18.723273 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:18.723273    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m994\" (UniqueName: \"kubernetes.io/projected/f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e-kube-api-access-9m994\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:37:19.159967 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:19.159929    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hk9qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 31 00:37:19.160161 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:19.159991    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf" podUID="a5f7f289-ef35-494d-91c9-034c7f1294d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 31 00:37:19.204646 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:19.204608    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4qhrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 31 00:37:19.205058 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:19.204668    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd" podUID="6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 31 00:37:19.303753 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:19.303713    4327 generic.go:356] "Generic (PLEG): container finished" podID="f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e" containerID="a9f3189289245033a953ff973f41a65e536c841652902d0307168aa143e1b5f3" exitCode=0
Oct 31 00:37:19.303973 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:19.303759    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-74d4469f5d-6vq6q" event={"ID":"f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e","Type":"ContainerDied","Data":"a9f3189289245033a953ff973f41a65e536c841652902d0307168aa143e1b5f3"}
Oct 31 00:37:19.303973 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:19.303827    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-74d4469f5d-6vq6q" event={"ID":"f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e","Type":"ContainerDied","Data":"6e3663c986e295d2f1da6ec1d7ebf026bf5c8c39e2a26242ca98d38008324863"}
Oct 31 00:37:19.303973 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:19.303778    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-74d4469f5d-6vq6q"
Oct 31 00:37:19.303973 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:19.303868    4327 scope.go:122] "RemoveContainer" containerID="a9f3189289245033a953ff973f41a65e536c841652902d0307168aa143e1b5f3"
Oct 31 00:37:19.314736 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:19.314708    4327 scope.go:122] "RemoveContainer" containerID="a9f3189289245033a953ff973f41a65e536c841652902d0307168aa143e1b5f3"
Oct 31 00:37:19.315046 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:37:19.315025    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9f3189289245033a953ff973f41a65e536c841652902d0307168aa143e1b5f3\": container with ID starting with a9f3189289245033a953ff973f41a65e536c841652902d0307168aa143e1b5f3 not found: ID does not exist" containerID="a9f3189289245033a953ff973f41a65e536c841652902d0307168aa143e1b5f3"
Oct 31 00:37:19.315135 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:19.315053    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9f3189289245033a953ff973f41a65e536c841652902d0307168aa143e1b5f3"} err="failed to get container status \"a9f3189289245033a953ff973f41a65e536c841652902d0307168aa143e1b5f3\": rpc error: code = NotFound desc = could not find container \"a9f3189289245033a953ff973f41a65e536c841652902d0307168aa143e1b5f3\": container with ID starting with a9f3189289245033a953ff973f41a65e536c841652902d0307168aa143e1b5f3 not found: ID does not exist"
Oct 31 00:37:19.357022 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:19.356985    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-74d4469f5d-6vq6q"]
Oct 31 00:37:19.369400 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:19.369361    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-74d4469f5d-6vq6q"]
Oct 31 00:37:19.927690 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:19.927632    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e" path="/var/lib/kubelet/pods/f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e/volumes"
Oct 31 00:37:20.940148 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:20.940109    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:20.940962 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:20.940934    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:20.941829 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:20.941791    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 00:37:20.989702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:20.988831    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:20.989702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:20.989468    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:20.990238 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:20.990213    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 00:37:45.293027 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:37:45.292988    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:37:46.565251 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.565211    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"]
Oct 31 00:37:46.566164 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.565990    4327 cpu_manager.go:414] "containerMap: removing container" podUID="f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e" containerName="manager"
Oct 31 00:37:46.566164 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.566010    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e" containerName="manager"
Oct 31 00:37:46.566164 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.566105    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9e3b76d-bfd2-4eaf-a6f8-f36b90a6919e" containerName="manager"
Oct 31 00:37:46.576515 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.576492    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.594236 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.593990    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b024b5-openshift-default-dockercfg-gwjft\""
Oct 31 00:37:46.608534 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.608496    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"]
Oct 31 00:37:46.682959 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.682920    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.683148 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.682978    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a24c7c0a-782b-4d04-976d-9f62a964250d-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.683148 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.683086    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.683148 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.683119    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.683148 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.683139    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.683305 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.683284    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.683343 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.683331    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.683376 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.683355    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.683376 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.683371    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9wkj\" (UniqueName: \"kubernetes.io/projected/a24c7c0a-782b-4d04-976d-9f62a964250d-kube-api-access-g9wkj\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.698625 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.698592    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"]
Oct 31 00:37:46.702295 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.702274    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.722878 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.722830    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-735123-openshift-default-dockercfg-z4chw\""
Oct 31 00:37:46.728464 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.728432    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"]
Oct 31 00:37:46.784842 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.784802    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.784842 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.784840    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.785163 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.784860    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.785163 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.784901    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.785163 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.784933    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.785163 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.785080    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.785163 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.785120    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.785436 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.785169    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.785436 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.785195    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9wkj\" (UniqueName: \"kubernetes.io/projected/a24c7c0a-782b-4d04-976d-9f62a964250d-kube-api-access-g9wkj\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.785436 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.785249    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.785436 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.785255    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.785436 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.785330    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.785436 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.785377    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.785731 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.785465    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gztr\" (UniqueName: \"kubernetes.io/projected/14093068-2225-44ec-9ec6-9f85d5beeb4f-kube-api-access-9gztr\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.785731 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.785503    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.785731 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.785521    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.785731 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.785535    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a24c7c0a-782b-4d04-976d-9f62a964250d-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.785731 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.785551    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.785731 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.785592    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14093068-2225-44ec-9ec6-9f85d5beeb4f-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.785731 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.785716    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.786005 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.785768    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.786005 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.785843    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.786258 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.786237    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a24c7c0a-782b-4d04-976d-9f62a964250d-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.787305 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.787285    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.787946 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.787931    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.811598 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.811562    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.811795 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.811745    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9wkj\" (UniqueName: \"kubernetes.io/projected/a24c7c0a-782b-4d04-976d-9f62a964250d-kube-api-access-g9wkj\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.835955 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.835870    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"]
Oct 31 00:37:46.841407 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.841376    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:46.857885 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.857858    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-daaf26-openshift-default-dockercfg-ld79m\""
Oct 31 00:37:46.868200 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.867038    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"]
Oct 31 00:37:46.887304 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.887269    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.887506 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.887319    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.887506 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.887373    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.887506 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.887409    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gztr\" (UniqueName: \"kubernetes.io/projected/14093068-2225-44ec-9ec6-9f85d5beeb4f-kube-api-access-9gztr\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.887506 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.887439    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.887506 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.887469    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14093068-2225-44ec-9ec6-9f85d5beeb4f-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.887506 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.887503    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.888214 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.887550    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.888214 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.887578    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.888389 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.888366    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.888737 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.888643    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.888925 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.888899    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14093068-2225-44ec-9ec6-9f85d5beeb4f-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.889283 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.889265    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.889464 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.889449    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.890714 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.890652    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.891588 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.891569    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:46.892943 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.892809    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.911713 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.911669    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.911838 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.911763    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gztr\" (UniqueName: \"kubernetes.io/projected/14093068-2225-44ec-9ec6-9f85d5beeb4f-kube-api-access-9gztr\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:46.989226 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.989188    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:46.989495 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.989469    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:46.989637 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.989620    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d93e1303-7225-4f18-b344-ffeea15da3b5-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:46.989798 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.989782    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:46.989939 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.989924    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkpkz\" (UniqueName: \"kubernetes.io/projected/d93e1303-7225-4f18-b344-ffeea15da3b5-kube-api-access-pkpkz\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:46.990044 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.990030    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:46.990167 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.990154    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:46.990284 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.990275    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:46.990352 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.990341    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:46.991516 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.991489    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"]
Oct 31 00:37:46.996691 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:46.996662    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.016942 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.016914    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:47.024942 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.024907    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"]
Oct 31 00:37:47.040651 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.040604    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-c9e377-openshift-default-dockercfg-hqt4g\""
Oct 31 00:37:47.091624 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.091549    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d93e1303-7225-4f18-b344-ffeea15da3b5-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:47.091624 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.091596    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:47.091838 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.091631    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.091838 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.091663    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.091838 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.091712    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkpkz\" (UniqueName: \"kubernetes.io/projected/d93e1303-7225-4f18-b344-ffeea15da3b5-kube-api-access-pkpkz\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:47.091838 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.091738    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:47.091838 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.091781    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.091838 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.091818    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.092150 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.091853    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzxxl\" (UniqueName: \"kubernetes.io/projected/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-kube-api-access-kzxxl\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.092150 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.091891    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:47.092150 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.091928    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.092150 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.092076    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:47.092150 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.092105    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:47.092474 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.092449    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:47.092591 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.092572    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.092793 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.092754    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.092846 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.092799    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.092846 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.092834    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:47.092919 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.092884    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:47.093611 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.093062    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:47.094059 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.094033    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:47.094199 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.094179    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:47.094647 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.094624    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d93e1303-7225-4f18-b344-ffeea15da3b5-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:47.095004 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.094984    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:47.095278 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.095259    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:47.118485 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.118457    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:47.119825 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.119808    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkpkz\" (UniqueName: \"kubernetes.io/projected/d93e1303-7225-4f18-b344-ffeea15da3b5-kube-api-access-pkpkz\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:47.157186 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.157149    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:47.193988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.193945    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.194174 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.194020    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.194174 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.194048    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.194174 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.194118    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.194174 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.194144    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.194387 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.194183    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.194387 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.194218    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.194387 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.194252    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzxxl\" (UniqueName: \"kubernetes.io/projected/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-kube-api-access-kzxxl\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.194387 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.194306    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.194564 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.194490    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.194763 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.194738    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.194865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.194798    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.194993 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.194975    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.195068 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.195051    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.196655 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.196635    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.197140 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.197120    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.226129 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.225334    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.227403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.227345    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzxxl\" (UniqueName: \"kubernetes.io/projected/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-kube-api-access-kzxxl\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.229699 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.229612    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"]
Oct 31 00:37:47.240217 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.240179    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:37:47.240337 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.240271    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:37:47.240337 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.240319    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:37:47.313408 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.313376    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:47.366557 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.365912    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:37:47.366557 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.366000    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:37:47.366557 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.366071    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:37:47.370968 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.369280    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"]
Oct 31 00:37:47.435537 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.435356    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6" event={"ID":"14093068-2225-44ec-9ec6-9f85d5beeb4f","Type":"ContainerStarted","Data":"4e76d5e7824c01fb8535c790a26915f155f5a0ed253d4ceef7d4b985ea1405eb"}
Oct 31 00:37:47.437530 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.437263    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq" event={"ID":"a24c7c0a-782b-4d04-976d-9f62a964250d","Type":"ContainerStarted","Data":"eec57e7ca9466cfe7d8f9347b97438dcd961993c5c26bdc11a5ff5ff7a77a272"}
Oct 31 00:37:47.437530 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.437338    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq" event={"ID":"a24c7c0a-782b-4d04-976d-9f62a964250d","Type":"ContainerStarted","Data":"2676cb3206573b1c673f693f9f61badbd6707c7b7ee611a247bb98fc0f3d9469"}
Oct 31 00:37:47.478604 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.478573    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"]
Oct 31 00:37:47.479373 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:37:47.479349    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd93e1303_7225_4f18_b344_ffeea15da3b5.slice/crio-3e3b107947d93b88caccd421e947044889b0bdf05c2734ed941b8a16e9b95c08 WatchSource:0}: Error finding container 3e3b107947d93b88caccd421e947044889b0bdf05c2734ed941b8a16e9b95c08: Status 404 returned error can't find the container with id 3e3b107947d93b88caccd421e947044889b0bdf05c2734ed941b8a16e9b95c08
Oct 31 00:37:47.481181 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.481147    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:37:47.481279 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.481254    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:37:47.481341 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.481300    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:37:47.495996 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.495243    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq" podStartSLOduration=1.495222428 podStartE2EDuration="1.495222428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:47.493199031 +0000 UTC m=+1127.715234572" watchObservedRunningTime="2026-10-31 00:37:47.495222428 +0000 UTC m=+1127.717257971"
Oct 31 00:37:47.642604 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.641842    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"]
Oct 31 00:37:47.645420 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.645369    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:37:47.645533 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.645441    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:37:47.645533 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.645478    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:37:47.892635 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:47.892556    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:48.444377 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:48.444339    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx" event={"ID":"d93e1303-7225-4f18-b344-ffeea15da3b5","Type":"ContainerStarted","Data":"e39dc35f9909ff40d3b9b5da429e6637263a9c3ff9f0c156b6628c0d33a63bfd"}
Oct 31 00:37:48.447782 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:48.444568    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx" event={"ID":"d93e1303-7225-4f18-b344-ffeea15da3b5","Type":"ContainerStarted","Data":"3e3b107947d93b88caccd421e947044889b0bdf05c2734ed941b8a16e9b95c08"}
Oct 31 00:37:48.453433 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:48.453396    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6" event={"ID":"14093068-2225-44ec-9ec6-9f85d5beeb4f","Type":"ContainerStarted","Data":"e0d4e6a9ebc0909ff8a1f19cdac60d9c215a7804f55bfc40cfdef0a915dbcf8a"}
Oct 31 00:37:48.455037 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:48.455008    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9" event={"ID":"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c","Type":"ContainerStarted","Data":"c48de8d5b4670edd4e87a5b6da7325a7b65e59159f2771c8f9223b620b93c315"}
Oct 31 00:37:48.455191 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:48.455053    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9" event={"ID":"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c","Type":"ContainerStarted","Data":"70e5f1191b543710e4625b79b50ca3526286954718060fa334cb75d71c1a6caa"}
Oct 31 00:37:48.520169 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:48.520106    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx" podStartSLOduration=2.520091663 podStartE2EDuration="2.520091663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:48.518908105 +0000 UTC m=+1128.740943649" watchObservedRunningTime="2026-10-31 00:37:48.520091663 +0000 UTC m=+1128.742127204"
Oct 31 00:37:48.588189 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:48.588131    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9" podStartSLOduration=2.588113105 podStartE2EDuration="2.588113105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:48.586877128 +0000 UTC m=+1128.808912669" watchObservedRunningTime="2026-10-31 00:37:48.588113105 +0000 UTC m=+1128.810148648"
Oct 31 00:37:48.634561 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:48.634501    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6" podStartSLOduration=2.6344857360000002 podStartE2EDuration="2.634485736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:48.632234138 +0000 UTC m=+1128.854269679" watchObservedRunningTime="2026-10-31 00:37:48.634485736 +0000 UTC m=+1128.856521278"
Oct 31 00:37:48.926425 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:48.926390    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:49.017087 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:49.017055    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:49.023779 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:49.023752    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:49.157924 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:49.157883    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:49.163326 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:49.163300    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:49.313656 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:49.313565    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:49.318870 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:49.318843    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:49.460436 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:49.460389    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:49.460624 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:49.460462    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:49.460624 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:49.460489    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:49.460624 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:49.460511    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:49.461557 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:49.461521    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:37:49.461669 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:49.461589    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:37:49.461766 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:49.461735    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:37:49.461808 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:49.461793    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:37:51.554190 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:51.554155    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9"]
Oct 31 00:37:51.557787 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:51.557766    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9"
Oct 31 00:37:51.581854 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:51.581821    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-b024b5-serving-cert\""
Oct 31 00:37:51.588715 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:51.588154    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9"]
Oct 31 00:37:51.653794 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:51.653753    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00845e5-30d2-4bd8-89a1-67072e3fef48-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9\" (UID: \"f00845e5-30d2-4bd8-89a1-67072e3fef48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9"
Oct 31 00:37:51.654003 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:51.653904    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjhjg\" (UniqueName: \"kubernetes.io/projected/f00845e5-30d2-4bd8-89a1-67072e3fef48-kube-api-access-qjhjg\") pod \"maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9\" (UID: \"f00845e5-30d2-4bd8-89a1-67072e3fef48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9"
Oct 31 00:37:51.654003 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:51.653933    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f00845e5-30d2-4bd8-89a1-67072e3fef48-metrics-certs\") pod \"maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9\" (UID: \"f00845e5-30d2-4bd8-89a1-67072e3fef48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9"
Oct 31 00:37:51.654003 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:51.653970    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f00845e5-30d2-4bd8-89a1-67072e3fef48-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9\" (UID: \"f00845e5-30d2-4bd8-89a1-67072e3fef48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9"
Oct 31 00:37:51.754698 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:51.754651    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00845e5-30d2-4bd8-89a1-67072e3fef48-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9\" (UID: \"f00845e5-30d2-4bd8-89a1-67072e3fef48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9"
Oct 31 00:37:51.754909 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:51.754793    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjhjg\" (UniqueName: \"kubernetes.io/projected/f00845e5-30d2-4bd8-89a1-67072e3fef48-kube-api-access-qjhjg\") pod \"maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9\" (UID: \"f00845e5-30d2-4bd8-89a1-67072e3fef48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9"
Oct 31 00:37:51.754909 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:51.754818    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f00845e5-30d2-4bd8-89a1-67072e3fef48-metrics-certs\") pod \"maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9\" (UID: \"f00845e5-30d2-4bd8-89a1-67072e3fef48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9"
Oct 31 00:37:51.754909 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:51.754844    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f00845e5-30d2-4bd8-89a1-67072e3fef48-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9\" (UID: \"f00845e5-30d2-4bd8-89a1-67072e3fef48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9"
Oct 31 00:37:51.755432 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:51.755400    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00845e5-30d2-4bd8-89a1-67072e3fef48-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9\" (UID: \"f00845e5-30d2-4bd8-89a1-67072e3fef48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9"
Oct 31 00:37:51.757725 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:51.757702    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f00845e5-30d2-4bd8-89a1-67072e3fef48-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9\" (UID: \"f00845e5-30d2-4bd8-89a1-67072e3fef48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9"
Oct 31 00:37:51.758125 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:51.758107    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f00845e5-30d2-4bd8-89a1-67072e3fef48-metrics-certs\") pod \"maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9\" (UID: \"f00845e5-30d2-4bd8-89a1-67072e3fef48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9"
Oct 31 00:37:51.779826 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:51.779790    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjhjg\" (UniqueName: \"kubernetes.io/projected/f00845e5-30d2-4bd8-89a1-67072e3fef48-kube-api-access-qjhjg\") pod \"maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9\" (UID: \"f00845e5-30d2-4bd8-89a1-67072e3fef48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9"
Oct 31 00:37:51.872139 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:51.872096    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9"
Oct 31 00:37:52.174737 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:52.174705    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9"]
Oct 31 00:37:52.176079 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:37:52.176051    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf00845e5_30d2_4bd8_89a1_67072e3fef48.slice/crio-605b8a6b10916d9c6f568dd53c821c26bb9acbe3cdc4eab7501d7a481ad62768 WatchSource:0}: Error finding container 605b8a6b10916d9c6f568dd53c821c26bb9acbe3cdc4eab7501d7a481ad62768: Status 404 returned error can't find the container with id 605b8a6b10916d9c6f568dd53c821c26bb9acbe3cdc4eab7501d7a481ad62768
Oct 31 00:37:52.474532 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:52.474456    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9" event={"ID":"f00845e5-30d2-4bd8-89a1-67072e3fef48","Type":"ContainerStarted","Data":"605b8a6b10916d9c6f568dd53c821c26bb9acbe3cdc4eab7501d7a481ad62768"}
Oct 31 00:37:53.480869 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:53.480833    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9" event={"ID":"f00845e5-30d2-4bd8-89a1-67072e3fef48","Type":"ContainerStarted","Data":"df5c3bc45463b245ce3fb874c96acde46bcd528b1d2c34f42acc8795ab6816d8"}
Oct 31 00:37:53.481295 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:53.480957    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9"
Oct 31 00:37:53.532556 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:53.532491    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9" podStartSLOduration=2.214461293 podStartE2EDuration="2.532476246s" totalImagesPullingTime="318.014953ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:53.530827706 +0000 UTC m=+1133.752863250" watchObservedRunningTime="2026-10-31 00:37:53.532476246 +0000 UTC m=+1133.754511787"
Oct 31 00:37:55.833217 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:55.833179    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl"]
Oct 31 00:37:55.837044 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:55.837024    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl"
Oct 31 00:37:55.857570 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:55.857534    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-735123-serving-cert\""
Oct 31 00:37:55.865007 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:55.864974    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl"]
Oct 31 00:37:55.911308 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:55.911275    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-maas-api-tls\") pod \"maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl\" (UID: \"eb06a1d3-494c-4b1b-bb35-30bbc7b23357\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl"
Oct 31 00:37:55.911488 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:55.911324    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl2cl\" (UniqueName: \"kubernetes.io/projected/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-kube-api-access-jl2cl\") pod \"maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl\" (UID: \"eb06a1d3-494c-4b1b-bb35-30bbc7b23357\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl"
Oct 31 00:37:55.911488 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:55.911456    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl\" (UID: \"eb06a1d3-494c-4b1b-bb35-30bbc7b23357\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl"
Oct 31 00:37:55.911597 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:55.911515    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-metrics-certs\") pod \"maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl\" (UID: \"eb06a1d3-494c-4b1b-bb35-30bbc7b23357\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl"
Oct 31 00:37:56.012925 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:56.012890    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-maas-api-tls\") pod \"maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl\" (UID: \"eb06a1d3-494c-4b1b-bb35-30bbc7b23357\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl"
Oct 31 00:37:56.013170 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:56.013136    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl2cl\" (UniqueName: \"kubernetes.io/projected/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-kube-api-access-jl2cl\") pod \"maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl\" (UID: \"eb06a1d3-494c-4b1b-bb35-30bbc7b23357\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl"
Oct 31 00:37:56.013280 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:56.013232    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl\" (UID: \"eb06a1d3-494c-4b1b-bb35-30bbc7b23357\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl"
Oct 31 00:37:56.013383 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:56.013297    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-metrics-certs\") pod \"maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl\" (UID: \"eb06a1d3-494c-4b1b-bb35-30bbc7b23357\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl"
Oct 31 00:37:56.014406 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:56.014380    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl\" (UID: \"eb06a1d3-494c-4b1b-bb35-30bbc7b23357\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl"
Oct 31 00:37:56.016590 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:56.016565    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-maas-api-tls\") pod \"maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl\" (UID: \"eb06a1d3-494c-4b1b-bb35-30bbc7b23357\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl"
Oct 31 00:37:56.016728 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:56.016616    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-metrics-certs\") pod \"maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl\" (UID: \"eb06a1d3-494c-4b1b-bb35-30bbc7b23357\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl"
Oct 31 00:37:56.039509 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:56.039479    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl2cl\" (UniqueName: \"kubernetes.io/projected/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-kube-api-access-jl2cl\") pod \"maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl\" (UID: \"eb06a1d3-494c-4b1b-bb35-30bbc7b23357\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl"
Oct 31 00:37:56.150924 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:56.150880    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl"
Oct 31 00:37:56.525318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:56.525279    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl"]
Oct 31 00:37:56.525606 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:37:56.525583    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb06a1d3_494c_4b1b_bb35_30bbc7b23357.slice/crio-8b4c935c9746aa14a0e4d839b9a5fc711db54e9c14a31ab6611e8dedcf1c446c WatchSource:0}: Error finding container 8b4c935c9746aa14a0e4d839b9a5fc711db54e9c14a31ab6611e8dedcf1c446c: Status 404 returned error can't find the container with id 8b4c935c9746aa14a0e4d839b9a5fc711db54e9c14a31ab6611e8dedcf1c446c
Oct 31 00:37:57.502647 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:57.502607    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl" event={"ID":"eb06a1d3-494c-4b1b-bb35-30bbc7b23357","Type":"ContainerStarted","Data":"90788b13355f851b3ee6bbcb601ad9f7060abefc8ff5c93ecaced51549db9d2a"}
Oct 31 00:37:57.503074 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:57.502671    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl" event={"ID":"eb06a1d3-494c-4b1b-bb35-30bbc7b23357","Type":"ContainerStarted","Data":"8b4c935c9746aa14a0e4d839b9a5fc711db54e9c14a31ab6611e8dedcf1c446c"}
Oct 31 00:37:57.503074 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:57.502843    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl"
Oct 31 00:37:57.571156 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:57.571091    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl" podStartSLOduration=2.109235146 podStartE2EDuration="2.571071028s" totalImagesPullingTime="461.835882ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:57.568948838 +0000 UTC m=+1137.790984381" watchObservedRunningTime="2026-10-31 00:37:57.571071028 +0000 UTC m=+1137.793106573"
Oct 31 00:37:57.973212 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:57.973172    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg"]
Oct 31 00:37:57.977020 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:57.976997    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg"
Oct 31 00:37:57.993388 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:57.993361    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-be301b-serving-cert\""
Oct 31 00:37:57.998975 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:57.998950    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg"]
Oct 31 00:37:58.073015 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:58.072898    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fac0cd38-95b3-4875-b81f-3e04f000f4b3-metrics-certs\") pod \"maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg\" (UID: \"fac0cd38-95b3-4875-b81f-3e04f000f4b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg"
Oct 31 00:37:58.073273 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:58.073083    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fac0cd38-95b3-4875-b81f-3e04f000f4b3-maas-api-tls\") pod \"maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg\" (UID: \"fac0cd38-95b3-4875-b81f-3e04f000f4b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg"
Oct 31 00:37:58.073405 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:58.073271    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q4xm\" (UniqueName: \"kubernetes.io/projected/fac0cd38-95b3-4875-b81f-3e04f000f4b3-kube-api-access-5q4xm\") pod \"maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg\" (UID: \"fac0cd38-95b3-4875-b81f-3e04f000f4b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg"
Oct 31 00:37:58.073498 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:58.073454    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fac0cd38-95b3-4875-b81f-3e04f000f4b3-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg\" (UID: \"fac0cd38-95b3-4875-b81f-3e04f000f4b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg"
Oct 31 00:37:58.176715 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:58.175284    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fac0cd38-95b3-4875-b81f-3e04f000f4b3-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg\" (UID: \"fac0cd38-95b3-4875-b81f-3e04f000f4b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg"
Oct 31 00:37:58.176715 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:58.175387    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fac0cd38-95b3-4875-b81f-3e04f000f4b3-metrics-certs\") pod \"maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg\" (UID: \"fac0cd38-95b3-4875-b81f-3e04f000f4b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg"
Oct 31 00:37:58.176715 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:58.175435    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fac0cd38-95b3-4875-b81f-3e04f000f4b3-maas-api-tls\") pod \"maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg\" (UID: \"fac0cd38-95b3-4875-b81f-3e04f000f4b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg"
Oct 31 00:37:58.176715 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:58.175549    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q4xm\" (UniqueName: \"kubernetes.io/projected/fac0cd38-95b3-4875-b81f-3e04f000f4b3-kube-api-access-5q4xm\") pod \"maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg\" (UID: \"fac0cd38-95b3-4875-b81f-3e04f000f4b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg"
Oct 31 00:37:58.177184 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:58.177161    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fac0cd38-95b3-4875-b81f-3e04f000f4b3-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg\" (UID: \"fac0cd38-95b3-4875-b81f-3e04f000f4b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg"
Oct 31 00:37:58.183376 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:58.183344    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fac0cd38-95b3-4875-b81f-3e04f000f4b3-metrics-certs\") pod \"maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg\" (UID: \"fac0cd38-95b3-4875-b81f-3e04f000f4b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg"
Oct 31 00:37:58.183649 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:58.183631    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fac0cd38-95b3-4875-b81f-3e04f000f4b3-maas-api-tls\") pod \"maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg\" (UID: \"fac0cd38-95b3-4875-b81f-3e04f000f4b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg"
Oct 31 00:37:58.200137 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:58.200101    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q4xm\" (UniqueName: \"kubernetes.io/projected/fac0cd38-95b3-4875-b81f-3e04f000f4b3-kube-api-access-5q4xm\") pod \"maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg\" (UID: \"fac0cd38-95b3-4875-b81f-3e04f000f4b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg"
Oct 31 00:37:58.290804 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:58.290705    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg"
Oct 31 00:37:58.684785 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:58.682167    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg"]
Oct 31 00:37:58.692225 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:58.692199    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl"]
Oct 31 00:37:58.702915 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:58.702886    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl"]
Oct 31 00:37:58.703202 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:58.703187    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl"
Oct 31 00:37:58.737743 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:58.727378    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-c9e377\""
Oct 31 00:37:58.787505 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:58.787464    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p242r\" (UniqueName: \"kubernetes.io/projected/eb16adc7-12a2-4497-a1a5-ad9adbe96684-kube-api-access-p242r\") pod \"payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl\" (UID: \"eb16adc7-12a2-4497-a1a5-ad9adbe96684\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl"
Oct 31 00:37:58.787671 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:58.787613    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb16adc7-12a2-4497-a1a5-ad9adbe96684-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl\" (UID: \"eb16adc7-12a2-4497-a1a5-ad9adbe96684\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl"
Oct 31 00:37:58.789753 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:58.789723    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-c9e377-dockercfg-n7h9c\""
Oct 31 00:37:58.888256 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:58.888208    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p242r\" (UniqueName: \"kubernetes.io/projected/eb16adc7-12a2-4497-a1a5-ad9adbe96684-kube-api-access-p242r\") pod \"payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl\" (UID: \"eb16adc7-12a2-4497-a1a5-ad9adbe96684\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl"
Oct 31 00:37:58.888472 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:58.888326    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb16adc7-12a2-4497-a1a5-ad9adbe96684-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl\" (UID: \"eb16adc7-12a2-4497-a1a5-ad9adbe96684\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl"
Oct 31 00:37:58.889041 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:58.889022    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb16adc7-12a2-4497-a1a5-ad9adbe96684-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl\" (UID: \"eb16adc7-12a2-4497-a1a5-ad9adbe96684\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl"
Oct 31 00:37:58.911993 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:58.911964    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p242r\" (UniqueName: \"kubernetes.io/projected/eb16adc7-12a2-4497-a1a5-ad9adbe96684-kube-api-access-p242r\") pod \"payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl\" (UID: \"eb16adc7-12a2-4497-a1a5-ad9adbe96684\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl"
Oct 31 00:37:59.073761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:59.073073    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl"
Oct 31 00:37:59.414428 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:59.414405    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl"]
Oct 31 00:37:59.416170 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:37:59.416143    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb16adc7_12a2_4497_a1a5_ad9adbe96684.slice/crio-3bbafea85e629c12d5e33b8b8f931d2a32810357ac38d147dda59b7d635eb8fc WatchSource:0}: Error finding container 3bbafea85e629c12d5e33b8b8f931d2a32810357ac38d147dda59b7d635eb8fc: Status 404 returned error can't find the container with id 3bbafea85e629c12d5e33b8b8f931d2a32810357ac38d147dda59b7d635eb8fc
Oct 31 00:37:59.491956 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:59.491938    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9"
Oct 31 00:37:59.514196 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:59.514152    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg" event={"ID":"fac0cd38-95b3-4875-b81f-3e04f000f4b3","Type":"ContainerStarted","Data":"da2739efeca4b39db30e2172bcc2fae27a02936e1bcbb4f13d6926a1307e1ab7"}
Oct 31 00:37:59.514363 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:59.514224    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg"
Oct 31 00:37:59.514363 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:59.514245    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg" event={"ID":"fac0cd38-95b3-4875-b81f-3e04f000f4b3","Type":"ContainerStarted","Data":"1656a38b920a021538a911880ef87ea7611035ffc3af5e9c4a74e8afd00c6873"}
Oct 31 00:37:59.515990 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:59.515961    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl" event={"ID":"eb16adc7-12a2-4497-a1a5-ad9adbe96684","Type":"ContainerStarted","Data":"a6998cf36be5fa6d73ae29c7c5738ce83f83d2f104d527ca164742774eefb85f"}
Oct 31 00:37:59.516107 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:59.516013    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl" event={"ID":"eb16adc7-12a2-4497-a1a5-ad9adbe96684","Type":"ContainerStarted","Data":"3bbafea85e629c12d5e33b8b8f931d2a32810357ac38d147dda59b7d635eb8fc"}
Oct 31 00:37:59.516107 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:59.516039    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl"
Oct 31 00:37:59.599433 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:37:59.599378    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl" podStartSLOduration=1.5993643629999998 podStartE2EDuration="1.599364363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:59.596250146 +0000 UTC m=+1139.818285712" watchObservedRunningTime="2026-10-31 00:37:59.599364363 +0000 UTC m=+1139.821399905"
Oct 31 00:38:00.346748 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:00.346618    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg" podStartSLOduration=2.884978028 podStartE2EDuration="3.346601018s" totalImagesPullingTime="461.62299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:59.644015466 +0000 UTC m=+1139.866051008" watchObservedRunningTime="2026-10-31 00:38:00.346601018 +0000 UTC m=+1140.568636617"
Oct 31 00:38:00.348024 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:00.348004    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67"]
Oct 31 00:38:00.360374 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:00.360346    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67"
Oct 31 00:38:00.374631 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:00.374598    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67"]
Oct 31 00:38:00.385484 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:00.385457    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-c9e377-serving-cert\""
Oct 31 00:38:00.408204 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:00.408158    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32310634-11e3-4d27-9fd4-e9c3af71002b-maas-api-tls\") pod \"maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67\" (UID: \"32310634-11e3-4d27-9fd4-e9c3af71002b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67"
Oct 31 00:38:00.408448 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:00.408344    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32310634-11e3-4d27-9fd4-e9c3af71002b-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67\" (UID: \"32310634-11e3-4d27-9fd4-e9c3af71002b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67"
Oct 31 00:38:00.408448 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:00.408385    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32310634-11e3-4d27-9fd4-e9c3af71002b-metrics-certs\") pod \"maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67\" (UID: \"32310634-11e3-4d27-9fd4-e9c3af71002b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67"
Oct 31 00:38:00.408605 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:00.408451    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl7f6\" (UniqueName: \"kubernetes.io/projected/32310634-11e3-4d27-9fd4-e9c3af71002b-kube-api-access-nl7f6\") pod \"maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67\" (UID: \"32310634-11e3-4d27-9fd4-e9c3af71002b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67"
Oct 31 00:38:00.509765 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:00.509731    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32310634-11e3-4d27-9fd4-e9c3af71002b-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67\" (UID: \"32310634-11e3-4d27-9fd4-e9c3af71002b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67"
Oct 31 00:38:00.509936 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:00.509779    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32310634-11e3-4d27-9fd4-e9c3af71002b-metrics-certs\") pod \"maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67\" (UID: \"32310634-11e3-4d27-9fd4-e9c3af71002b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67"
Oct 31 00:38:00.509936 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:00.509799    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl7f6\" (UniqueName: \"kubernetes.io/projected/32310634-11e3-4d27-9fd4-e9c3af71002b-kube-api-access-nl7f6\") pod \"maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67\" (UID: \"32310634-11e3-4d27-9fd4-e9c3af71002b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67"
Oct 31 00:38:00.509936 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:00.509855    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32310634-11e3-4d27-9fd4-e9c3af71002b-maas-api-tls\") pod \"maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67\" (UID: \"32310634-11e3-4d27-9fd4-e9c3af71002b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67"
Oct 31 00:38:00.510134 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:38:00.510073    4327 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-serving-cert: secret "maas-api-e2e-worker-w5-c9e377-serving-cert" not found
Oct 31 00:38:00.510211 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:38:00.510161    4327 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32310634-11e3-4d27-9fd4-e9c3af71002b-maas-api-tls podName:32310634-11e3-4d27-9fd4-e9c3af71002b nodeName:}" failed. No retries permitted until 2026-10-31 00:38:01.010138546 +0000 UTC m=+1141.232174090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/32310634-11e3-4d27-9fd4-e9c3af71002b-maas-api-tls") pod "maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67" (UID: "32310634-11e3-4d27-9fd4-e9c3af71002b") : secret "maas-api-e2e-worker-w5-c9e377-serving-cert" not found
Oct 31 00:38:00.510574 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:00.510554    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32310634-11e3-4d27-9fd4-e9c3af71002b-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67\" (UID: \"32310634-11e3-4d27-9fd4-e9c3af71002b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67"
Oct 31 00:38:00.512266 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:00.512241    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32310634-11e3-4d27-9fd4-e9c3af71002b-metrics-certs\") pod \"maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67\" (UID: \"32310634-11e3-4d27-9fd4-e9c3af71002b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67"
Oct 31 00:38:00.550517 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:00.550488    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl7f6\" (UniqueName: \"kubernetes.io/projected/32310634-11e3-4d27-9fd4-e9c3af71002b-kube-api-access-nl7f6\") pod \"maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67\" (UID: \"32310634-11e3-4d27-9fd4-e9c3af71002b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67"
Oct 31 00:38:01.016227 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:01.016192    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32310634-11e3-4d27-9fd4-e9c3af71002b-maas-api-tls\") pod \"maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67\" (UID: \"32310634-11e3-4d27-9fd4-e9c3af71002b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67"
Oct 31 00:38:01.026086 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:01.021272    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32310634-11e3-4d27-9fd4-e9c3af71002b-maas-api-tls\") pod \"maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67\" (UID: \"32310634-11e3-4d27-9fd4-e9c3af71002b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67"
Oct 31 00:38:01.274100 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:01.274015    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67"
Oct 31 00:38:01.651495 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:01.651465    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67"]
Oct 31 00:38:02.766393 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:02.766290    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67" event={"ID":"32310634-11e3-4d27-9fd4-e9c3af71002b","Type":"ContainerStarted","Data":"e25238ea0e069c3d2c09c4bb944efc171a590abcfcc4d83a3c1fff719c5db323"}
Oct 31 00:38:03.513106 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:03.513011    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl"
Oct 31 00:38:03.774381 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:03.774298    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67" event={"ID":"32310634-11e3-4d27-9fd4-e9c3af71002b","Type":"ContainerStarted","Data":"ddfb934e3598b5879d065bbd8ce6618fd88a85d74d1e89d54320a9387ff8341c"}
Oct 31 00:38:03.775034 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:03.775017    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67"
Oct 31 00:38:04.595605 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:04.595545    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67" podStartSLOduration=4.0991926 podStartE2EDuration="4.595527871s" totalImagesPullingTime="496.335271ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:03.827665463 +0000 UTC m=+1144.049701006" watchObservedRunningTime="2026-10-31 00:38:04.595527871 +0000 UTC m=+1144.817563412"
Oct 31 00:38:04.596930 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:04.596806    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc"]
Oct 31 00:38:04.600597 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:04.600580    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc"
Oct 31 00:38:04.642703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:04.642037    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-0d8bae-serving-cert\""
Oct 31 00:38:04.689870 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:04.689818    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc"]
Oct 31 00:38:04.694770 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:04.691667    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-maas-api-tls\") pod \"maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc\" (UID: \"f39647e7-efed-4224-80d9-e7a1cfc9c8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc"
Oct 31 00:38:04.694770 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:04.691751    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc\" (UID: \"f39647e7-efed-4224-80d9-e7a1cfc9c8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc"
Oct 31 00:38:04.694770 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:04.691863    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-metrics-certs\") pod \"maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc\" (UID: \"f39647e7-efed-4224-80d9-e7a1cfc9c8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc"
Oct 31 00:38:04.694770 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:04.692040    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gv6h\" (UniqueName: \"kubernetes.io/projected/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-kube-api-access-4gv6h\") pod \"maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc\" (UID: \"f39647e7-efed-4224-80d9-e7a1cfc9c8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc"
Oct 31 00:38:04.794775 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:04.793977    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-maas-api-tls\") pod \"maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc\" (UID: \"f39647e7-efed-4224-80d9-e7a1cfc9c8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc"
Oct 31 00:38:04.794775 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:04.794042    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc\" (UID: \"f39647e7-efed-4224-80d9-e7a1cfc9c8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc"
Oct 31 00:38:04.794775 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:04.794146    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-metrics-certs\") pod \"maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc\" (UID: \"f39647e7-efed-4224-80d9-e7a1cfc9c8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc"
Oct 31 00:38:04.794775 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:38:04.794155    4327 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-serving-cert: secret "maas-api-e2e-worker-w4-0d8bae-serving-cert" not found
Oct 31 00:38:04.794775 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:04.794177    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gv6h\" (UniqueName: \"kubernetes.io/projected/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-kube-api-access-4gv6h\") pod \"maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc\" (UID: \"f39647e7-efed-4224-80d9-e7a1cfc9c8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc"
Oct 31 00:38:04.794775 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:38:04.794243    4327 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-maas-api-tls podName:f39647e7-efed-4224-80d9-e7a1cfc9c8a1 nodeName:}" failed. No retries permitted until 2026-10-31 00:38:05.294218734 +0000 UTC m=+1145.516254276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-maas-api-tls") pod "maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc" (UID: "f39647e7-efed-4224-80d9-e7a1cfc9c8a1") : secret "maas-api-e2e-worker-w4-0d8bae-serving-cert" not found
Oct 31 00:38:04.795299 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:04.794895    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc\" (UID: \"f39647e7-efed-4224-80d9-e7a1cfc9c8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc"
Oct 31 00:38:04.797362 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:04.797334    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-metrics-certs\") pod \"maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc\" (UID: \"f39647e7-efed-4224-80d9-e7a1cfc9c8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc"
Oct 31 00:38:04.881814 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:04.881781    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gv6h\" (UniqueName: \"kubernetes.io/projected/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-kube-api-access-4gv6h\") pod \"maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc\" (UID: \"f39647e7-efed-4224-80d9-e7a1cfc9c8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc"
Oct 31 00:38:05.300463 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:05.300331    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-maas-api-tls\") pod \"maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc\" (UID: \"f39647e7-efed-4224-80d9-e7a1cfc9c8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc"
Oct 31 00:38:05.303323 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:05.303295    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-maas-api-tls\") pod \"maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc\" (UID: \"f39647e7-efed-4224-80d9-e7a1cfc9c8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc"
Oct 31 00:38:05.513138 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:05.508882    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:38:05.513138 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:05.508954    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq" podUID="a24c7c0a-782b-4d04-976d-9f62a964250d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:38:05.517647 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:05.516975    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc"
Oct 31 00:38:05.588715 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:05.585916    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg"
Oct 31 00:38:05.979007 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:05.978921    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc"]
Oct 31 00:38:06.798452 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:06.798360    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc" event={"ID":"f39647e7-efed-4224-80d9-e7a1cfc9c8a1","Type":"ContainerStarted","Data":"821112320cef9412c0dcf8734d65bb1ec9ef2d04fde70e7b5ca04fcd02080ea6"}
Oct 31 00:38:06.798452 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:06.798415    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc" event={"ID":"f39647e7-efed-4224-80d9-e7a1cfc9c8a1","Type":"ContainerStarted","Data":"bd372d48f529ed8865623aa11f38fa421248f12d0f95ce9c8fc3d3598f7814d7"}
Oct 31 00:38:06.799154 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:06.799134    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc"
Oct 31 00:38:08.825232 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:08.825197    4327 generic.go:356] "Generic (PLEG): container finished" podID="a24c7c0a-782b-4d04-976d-9f62a964250d" containerID="eec57e7ca9466cfe7d8f9347b97438dcd961993c5c26bdc11a5ff5ff7a77a272" exitCode=0
Oct 31 00:38:08.825837 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:08.825274    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq" event={"ID":"a24c7c0a-782b-4d04-976d-9f62a964250d","Type":"ContainerDied","Data":"eec57e7ca9466cfe7d8f9347b97438dcd961993c5c26bdc11a5ff5ff7a77a272"}
Oct 31 00:38:08.825837 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:08.825781    4327 scope.go:122] "RemoveContainer" containerID="eec57e7ca9466cfe7d8f9347b97438dcd961993c5c26bdc11a5ff5ff7a77a272"
Oct 31 00:38:08.827580 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:08.827540    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:38:08.827667 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:08.827644    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:38:08.827750 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:08.827734    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:38:08.889106 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:08.889052    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc" podStartSLOduration=4.479276108 podStartE2EDuration="4.889037771s" totalImagesPullingTime="409.761663ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:06.848906345 +0000 UTC m=+1147.070941912" watchObservedRunningTime="2026-10-31 00:38:08.889037771 +0000 UTC m=+1149.111073313"
Oct 31 00:38:09.889984 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:09.889945    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq" event={"ID":"a24c7c0a-782b-4d04-976d-9f62a964250d","Type":"ContainerStarted","Data":"903646467c97419f3a1be016e12ed875f2c66114cde668fd93e6ce2eb34b22d3"}
Oct 31 00:38:10.565118 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:10.564071    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl"
Oct 31 00:38:10.809418 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:10.809385    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67"
Oct 31 00:38:10.892130 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:10.892103    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:38:10.892860 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:10.892314    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 31 00:38:10.893174 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:10.893077    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq" podUID="a24c7c0a-782b-4d04-976d-9f62a964250d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 31 00:38:11.893051 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:11.893008    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 31 00:38:11.893452 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:11.893091    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq" podUID="a24c7c0a-782b-4d04-976d-9f62a964250d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 31 00:38:12.893167 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:12.893052    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 31 00:38:12.893672 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:12.893141    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq" podUID="a24c7c0a-782b-4d04-976d-9f62a964250d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 31 00:38:13.868094 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:13.866976    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc"
Oct 31 00:38:13.892768 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:13.892739    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 31 00:38:13.893728 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:13.892800    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq" podUID="a24c7c0a-782b-4d04-976d-9f62a964250d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 31 00:38:13.983734 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:13.983667    4327 generic.go:356] "Generic (PLEG): container finished" podID="d93e1303-7225-4f18-b344-ffeea15da3b5" containerID="e39dc35f9909ff40d3b9b5da429e6637263a9c3ff9f0c156b6628c0d33a63bfd" exitCode=0
Oct 31 00:38:13.983913 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:13.983855    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx" event={"ID":"d93e1303-7225-4f18-b344-ffeea15da3b5","Type":"ContainerDied","Data":"e39dc35f9909ff40d3b9b5da429e6637263a9c3ff9f0c156b6628c0d33a63bfd"}
Oct 31 00:38:13.984337 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:13.984318    4327 scope.go:122] "RemoveContainer" containerID="e39dc35f9909ff40d3b9b5da429e6637263a9c3ff9f0c156b6628c0d33a63bfd"
Oct 31 00:38:13.985959 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:13.985921    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:38:13.986033 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:13.986016    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:38:13.986076 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:13.986066    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:38:14.893227 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:14.893186    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 31 00:38:14.893394 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:14.893254    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq" podUID="a24c7c0a-782b-4d04-976d-9f62a964250d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 31 00:38:14.994449 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:14.994416    4327 generic.go:356] "Generic (PLEG): container finished" podID="14093068-2225-44ec-9ec6-9f85d5beeb4f" containerID="e0d4e6a9ebc0909ff8a1f19cdac60d9c215a7804f55bfc40cfdef0a915dbcf8a" exitCode=0
Oct 31 00:38:14.994892 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:14.994493    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6" event={"ID":"14093068-2225-44ec-9ec6-9f85d5beeb4f","Type":"ContainerDied","Data":"e0d4e6a9ebc0909ff8a1f19cdac60d9c215a7804f55bfc40cfdef0a915dbcf8a"}
Oct 31 00:38:14.994956 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:14.994900    4327 scope.go:122] "RemoveContainer" containerID="e0d4e6a9ebc0909ff8a1f19cdac60d9c215a7804f55bfc40cfdef0a915dbcf8a"
Oct 31 00:38:14.996301 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:14.996266    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:38:14.996429 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:14.996343    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:38:14.996429 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:14.996378    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:38:14.998133 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:14.998111    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx" event={"ID":"d93e1303-7225-4f18-b344-ffeea15da3b5","Type":"ContainerStarted","Data":"e540b830445d04d603f1c5ae3c06717d29902873a0c4413b8ddf7736ce5815fb"}
Oct 31 00:38:15.173052 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:15.172097    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:38:15.173789 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:15.173605    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 00:38:15.173789 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:15.173661    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx" podUID="d93e1303-7225-4f18-b344-ffeea15da3b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 00:38:15.364146 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:38:15.364101    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:38:15.910710 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:15.909499    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:38:15.910710 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:15.910418    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:38:15.911220 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:15.911203    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 00:38:16.006388 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:16.006351    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6" event={"ID":"14093068-2225-44ec-9ec6-9f85d5beeb4f","Type":"ContainerStarted","Data":"65fe431367a34165bbcc15c1b123d3ede06439585de31ecc2585fba2f00a5d4e"}
Oct 31 00:38:16.027803 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:16.027764    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:38:16.035778 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:16.029608    4327 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 00:38:16.035778 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:16.029664    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6" podUID="14093068-2225-44ec-9ec6-9f85d5beeb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 00:38:16.175065 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:16.174981    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 00:38:16.175065 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:16.175045    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx" podUID="d93e1303-7225-4f18-b344-ffeea15da3b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 00:38:17.034705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:17.032003    4327 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 00:38:17.034705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:17.032071    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6" podUID="14093068-2225-44ec-9ec6-9f85d5beeb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 00:38:17.165899 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:17.161108    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 00:38:17.165899 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:17.161182    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx" podUID="d93e1303-7225-4f18-b344-ffeea15da3b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 00:38:18.035708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:18.021866    4327 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 00:38:18.035708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:18.021973    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6" podUID="14093068-2225-44ec-9ec6-9f85d5beeb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 00:38:18.160311 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:18.160276    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 00:38:18.160590 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:18.160559    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx" podUID="d93e1303-7225-4f18-b344-ffeea15da3b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 00:38:19.031465 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:19.031429    4327 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 00:38:19.031778 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:19.031749    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6" podUID="14093068-2225-44ec-9ec6-9f85d5beeb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 00:38:19.176707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:19.163260    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 00:38:19.176707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:19.163326    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx" podUID="d93e1303-7225-4f18-b344-ffeea15da3b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 00:38:19.462704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:19.461780    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:38:19.467726 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:19.464302    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:38:20.041705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:20.021873    4327 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 00:38:20.042059 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:20.042011    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6" podUID="14093068-2225-44ec-9ec6-9f85d5beeb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 00:38:20.174639 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:20.174587    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 00:38:20.174873 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:20.174650    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx" podUID="d93e1303-7225-4f18-b344-ffeea15da3b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 00:38:20.503872 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:20.503824    4327 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:38:20.504299 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:20.503911    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9" podUID="1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:38:21.021491 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:21.021052    4327 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 00:38:21.021491 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:21.021116    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6" podUID="14093068-2225-44ec-9ec6-9f85d5beeb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 00:38:21.163336 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:21.163301    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:38:21.164393 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:21.164369    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 00:38:22.022700 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:22.022643    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:38:22.023825 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:22.023803    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:38:22.123022 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:22.122985    4327 generic.go:356] "Generic (PLEG): container finished" podID="1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c" containerID="c48de8d5b4670edd4e87a5b6da7325a7b65e59159f2771c8f9223b620b93c315" exitCode=0
Oct 31 00:38:22.123220 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:22.123058    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9" event={"ID":"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c","Type":"ContainerDied","Data":"c48de8d5b4670edd4e87a5b6da7325a7b65e59159f2771c8f9223b620b93c315"}
Oct 31 00:38:22.123812 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:22.123783    4327 scope.go:122] "RemoveContainer" containerID="c48de8d5b4670edd4e87a5b6da7325a7b65e59159f2771c8f9223b620b93c315"
Oct 31 00:38:22.125016 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:22.124988    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:38:22.125099 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:22.125058    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:38:22.125099 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:22.125090    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:38:23.159231 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:23.159201    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9" event={"ID":"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c","Type":"ContainerStarted","Data":"c5165caf7853cd703ac494fc85f1a4f5dd18a0cda7d66b1e4d497148e5135681"}
Oct 31 00:38:23.314054 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:23.314015    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:38:23.314322 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:23.314299    4327 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 31 00:38:23.314368 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:23.314348    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9" podUID="1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 31 00:38:24.314348 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:24.314307    4327 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 31 00:38:24.314941 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:24.314364    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9" podUID="1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 31 00:38:25.314260 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:25.314226    4327 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 31 00:38:25.314438 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:25.314314    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9" podUID="1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 31 00:38:26.314734 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:26.314662    4327 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 31 00:38:26.315133 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:26.314769    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9" podUID="1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 31 00:38:27.313999 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:27.313963    4327 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 31 00:38:27.314181 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:27.314026    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9" podUID="1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 31 00:38:28.406357 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:28.406322    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:38:28.406877 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:28.406721    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:38:28.407928 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:38:28.407903    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:38:45.292188 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:38:45.292149    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:39:15.292356 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:39:15.292265    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:39:18.860634 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:18.860600    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"]
Oct 31 00:39:18.864500 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:18.864475    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:39:18.879717 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:18.879674    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-c9e377\"/\"kube-root-ca.crt\""
Oct 31 00:39:18.891517 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:18.891484    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-c9e377\"/\"openshift-service-ca.crt\""
Oct 31 00:39:18.892468 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:18.892425    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c9e377\"/\"premium-w5-c9e377-kserve-self-signed-certs\""
Oct 31 00:39:18.893860 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:18.893836    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c9e377\"/\"default-dockercfg-m7xjg\""
Oct 31 00:39:18.897735 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:18.897702    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"]
Oct 31 00:39:19.000394 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:19.000353    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-dshm\") pod \"premium-w5-c9e377-kserve-856f7b986c-5sd99\" (UID: \"681c6060-3662-4f0e-b768-2414c4750db4\") " pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:39:19.000608 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:19.000420    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-model-cache\") pod \"premium-w5-c9e377-kserve-856f7b986c-5sd99\" (UID: \"681c6060-3662-4f0e-b768-2414c4750db4\") " pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:39:19.000608 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:19.000522    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfdt8\" (UniqueName: \"kubernetes.io/projected/681c6060-3662-4f0e-b768-2414c4750db4-kube-api-access-bfdt8\") pod \"premium-w5-c9e377-kserve-856f7b986c-5sd99\" (UID: \"681c6060-3662-4f0e-b768-2414c4750db4\") " pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:39:19.000608 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:19.000564    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-home\") pod \"premium-w5-c9e377-kserve-856f7b986c-5sd99\" (UID: \"681c6060-3662-4f0e-b768-2414c4750db4\") " pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:39:19.000768 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:19.000661    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-tmp-dir\") pod \"premium-w5-c9e377-kserve-856f7b986c-5sd99\" (UID: \"681c6060-3662-4f0e-b768-2414c4750db4\") " pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:39:19.000768 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:19.000720    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/681c6060-3662-4f0e-b768-2414c4750db4-tls-certs\") pod \"premium-w5-c9e377-kserve-856f7b986c-5sd99\" (UID: \"681c6060-3662-4f0e-b768-2414c4750db4\") " pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:39:19.102603 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:19.102562    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-dshm\") pod \"premium-w5-c9e377-kserve-856f7b986c-5sd99\" (UID: \"681c6060-3662-4f0e-b768-2414c4750db4\") " pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:39:19.102949 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:19.102924    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-model-cache\") pod \"premium-w5-c9e377-kserve-856f7b986c-5sd99\" (UID: \"681c6060-3662-4f0e-b768-2414c4750db4\") " pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:39:19.103119 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:19.103101    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfdt8\" (UniqueName: \"kubernetes.io/projected/681c6060-3662-4f0e-b768-2414c4750db4-kube-api-access-bfdt8\") pod \"premium-w5-c9e377-kserve-856f7b986c-5sd99\" (UID: \"681c6060-3662-4f0e-b768-2414c4750db4\") " pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:39:19.103255 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:19.103241    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-home\") pod \"premium-w5-c9e377-kserve-856f7b986c-5sd99\" (UID: \"681c6060-3662-4f0e-b768-2414c4750db4\") " pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:39:19.103388 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:19.103375    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-tmp-dir\") pod \"premium-w5-c9e377-kserve-856f7b986c-5sd99\" (UID: \"681c6060-3662-4f0e-b768-2414c4750db4\") " pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:39:19.103478 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:19.103467    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/681c6060-3662-4f0e-b768-2414c4750db4-tls-certs\") pod \"premium-w5-c9e377-kserve-856f7b986c-5sd99\" (UID: \"681c6060-3662-4f0e-b768-2414c4750db4\") " pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:39:19.104896 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:19.104866    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-model-cache\") pod \"premium-w5-c9e377-kserve-856f7b986c-5sd99\" (UID: \"681c6060-3662-4f0e-b768-2414c4750db4\") " pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:39:19.105612 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:19.105592    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-home\") pod \"premium-w5-c9e377-kserve-856f7b986c-5sd99\" (UID: \"681c6060-3662-4f0e-b768-2414c4750db4\") " pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:39:19.106080 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:19.106051    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-tmp-dir\") pod \"premium-w5-c9e377-kserve-856f7b986c-5sd99\" (UID: \"681c6060-3662-4f0e-b768-2414c4750db4\") " pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:39:19.109339 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:19.109311    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-dshm\") pod \"premium-w5-c9e377-kserve-856f7b986c-5sd99\" (UID: \"681c6060-3662-4f0e-b768-2414c4750db4\") " pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:39:19.109464 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:19.109420    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/681c6060-3662-4f0e-b768-2414c4750db4-tls-certs\") pod \"premium-w5-c9e377-kserve-856f7b986c-5sd99\" (UID: \"681c6060-3662-4f0e-b768-2414c4750db4\") " pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:39:19.129700 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:19.129610    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfdt8\" (UniqueName: \"kubernetes.io/projected/681c6060-3662-4f0e-b768-2414c4750db4-kube-api-access-bfdt8\") pod \"premium-w5-c9e377-kserve-856f7b986c-5sd99\" (UID: \"681c6060-3662-4f0e-b768-2414c4750db4\") " pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:39:19.177627 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:19.177590    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:39:19.475790 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:39:19.475746    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod681c6060_3662_4f0e_b768_2414c4750db4.slice/crio-8a63533ce05d72fbe67f689311d5f765c4e38fc8521d7e3a789f021002de4ac3 WatchSource:0}: Error finding container 8a63533ce05d72fbe67f689311d5f765c4e38fc8521d7e3a789f021002de4ac3: Status 404 returned error can't find the container with id 8a63533ce05d72fbe67f689311d5f765c4e38fc8521d7e3a789f021002de4ac3
Oct 31 00:39:19.476153 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:19.476131    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"]
Oct 31 00:39:20.485587 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:20.485545    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" event={"ID":"681c6060-3662-4f0e-b768-2414c4750db4","Type":"ContainerStarted","Data":"0abd506dbe8103d3eccdc3058f2ec9ee47e528bbf622b167fef7c8a488cfd9c9"}
Oct 31 00:39:20.486032 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:20.485596    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" event={"ID":"681c6060-3662-4f0e-b768-2414c4750db4","Type":"ContainerStarted","Data":"8a63533ce05d72fbe67f689311d5f765c4e38fc8521d7e3a789f021002de4ac3"}
Oct 31 00:39:20.530047 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:20.529990    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podStartSLOduration=2.529975125 podStartE2EDuration="2.529975125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:20.528068249 +0000 UTC m=+1220.750103817" watchObservedRunningTime="2026-10-31 00:39:20.529975125 +0000 UTC m=+1220.752010711"
Oct 31 00:39:21.177969 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:21.177931    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:39:28.010275 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.010240    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"]
Oct 31 00:39:28.015946 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.015888    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:39:28.034505 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.034473    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"]
Oct 31 00:39:28.044236 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.044193    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0d8bae\"/\"openshift-service-ca.crt\""
Oct 31 00:39:28.044414 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.044249    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0d8bae\"/\"kube-root-ca.crt\""
Oct 31 00:39:28.045199 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.045171    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0d8bae\"/\"default-dockercfg-422tf\""
Oct 31 00:39:28.045462 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.045435    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0d8bae\"/\"premium-w4-0d8bae-kserve-self-signed-certs\""
Oct 31 00:39:28.147189 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.147153    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-tmp-dir\") pod \"premium-w4-0d8bae-kserve-89f6c45cd-gxfpp\" (UID: \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\") " pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:39:28.147371 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.147194    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt2gx\" (UniqueName: \"kubernetes.io/projected/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-kube-api-access-xt2gx\") pod \"premium-w4-0d8bae-kserve-89f6c45cd-gxfpp\" (UID: \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\") " pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:39:28.147371 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.147352    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-home\") pod \"premium-w4-0d8bae-kserve-89f6c45cd-gxfpp\" (UID: \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\") " pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:39:28.147444 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.147402    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-dshm\") pod \"premium-w4-0d8bae-kserve-89f6c45cd-gxfpp\" (UID: \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\") " pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:39:28.147498 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.147486    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-tls-certs\") pod \"premium-w4-0d8bae-kserve-89f6c45cd-gxfpp\" (UID: \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\") " pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:39:28.147534 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.147527    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-model-cache\") pod \"premium-w4-0d8bae-kserve-89f6c45cd-gxfpp\" (UID: \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\") " pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:39:28.249417 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.248277    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-dshm\") pod \"premium-w4-0d8bae-kserve-89f6c45cd-gxfpp\" (UID: \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\") " pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:39:28.249417 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.248355    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-tls-certs\") pod \"premium-w4-0d8bae-kserve-89f6c45cd-gxfpp\" (UID: \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\") " pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:39:28.249417 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.248420    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-model-cache\") pod \"premium-w4-0d8bae-kserve-89f6c45cd-gxfpp\" (UID: \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\") " pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:39:28.249417 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.248458    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-tmp-dir\") pod \"premium-w4-0d8bae-kserve-89f6c45cd-gxfpp\" (UID: \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\") " pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:39:28.249417 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.248489    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt2gx\" (UniqueName: \"kubernetes.io/projected/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-kube-api-access-xt2gx\") pod \"premium-w4-0d8bae-kserve-89f6c45cd-gxfpp\" (UID: \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\") " pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:39:28.249417 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.248608    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-home\") pod \"premium-w4-0d8bae-kserve-89f6c45cd-gxfpp\" (UID: \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\") " pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:39:28.249417 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.249030    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-home\") pod \"premium-w4-0d8bae-kserve-89f6c45cd-gxfpp\" (UID: \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\") " pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:39:28.250650 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.250329    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-model-cache\") pod \"premium-w4-0d8bae-kserve-89f6c45cd-gxfpp\" (UID: \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\") " pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:39:28.251106 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.251085    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-tmp-dir\") pod \"premium-w4-0d8bae-kserve-89f6c45cd-gxfpp\" (UID: \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\") " pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:39:28.254244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.254217    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-dshm\") pod \"premium-w4-0d8bae-kserve-89f6c45cd-gxfpp\" (UID: \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\") " pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:39:28.256752 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.256725    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-tls-certs\") pod \"premium-w4-0d8bae-kserve-89f6c45cd-gxfpp\" (UID: \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\") " pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:39:28.274180 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.274110    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt2gx\" (UniqueName: \"kubernetes.io/projected/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-kube-api-access-xt2gx\") pod \"premium-w4-0d8bae-kserve-89f6c45cd-gxfpp\" (UID: \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\") " pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:39:28.330094 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.330044    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:39:28.667027 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:28.666893    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"]
Oct 31 00:39:29.178361 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:29.178313    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:39:29.192594 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:29.192566    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:39:29.545025 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:29.544930    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" event={"ID":"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de","Type":"ContainerStarted","Data":"fc317fe8176d758f56bae6909ee2bb84fe588da7e3f6dc6eae6e147fda9ab8ba"}
Oct 31 00:39:29.545025 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:29.544977    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" event={"ID":"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de","Type":"ContainerStarted","Data":"188457dac756358d6626a35d22f4c298cfb2e1d20f77adee5ea5703cbb5c610f"}
Oct 31 00:39:29.556353 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:29.556322    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:39:29.587314 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:29.587262    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podStartSLOduration=2.587246759 podStartE2EDuration="2.587246759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:29.58648091 +0000 UTC m=+1229.808516453" watchObservedRunningTime="2026-10-31 00:39:29.587246759 +0000 UTC m=+1229.809282301"
Oct 31 00:39:30.330650 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:30.330617    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:39:38.330932 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:38.330891    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:39:38.344922 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:38.344888    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:39:38.603248 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:39:38.603220    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:39:45.292912 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:39:45.292871    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:40:02.652296 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.652255    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"]
Oct 31 00:40:02.657765 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.657740    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.672746 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.672718    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-671959-openshift-default-dockercfg-mgz7q\""
Oct 31 00:40:02.687655 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.687619    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"]
Oct 31 00:40:02.790622 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.790581    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.790833 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.790646    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.790833 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.790766    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.790833 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.790795    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.790833 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.790814    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbb151f9-f859-4faf-a830-9da6730837c6-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.790833 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.790833    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbnf5\" (UniqueName: \"kubernetes.io/projected/dbb151f9-f859-4faf-a830-9da6730837c6-kube-api-access-xbnf5\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.791000 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.790920    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbb151f9-f859-4faf-a830-9da6730837c6-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.791000 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.790962    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.791000 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.790980    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbb151f9-f859-4faf-a830-9da6730837c6-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.892206 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.892159    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.892206 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.892209    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbb151f9-f859-4faf-a830-9da6730837c6-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.892505 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.892240    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbnf5\" (UniqueName: \"kubernetes.io/projected/dbb151f9-f859-4faf-a830-9da6730837c6-kube-api-access-xbnf5\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.892505 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.892290    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbb151f9-f859-4faf-a830-9da6730837c6-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.892505 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.892329    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.892505 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.892351    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbb151f9-f859-4faf-a830-9da6730837c6-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.892750 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.892517    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.892750 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.892591    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.892750 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.892657    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.892905 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.892653    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.892905 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.892881    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.893146 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.893118    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.893235 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.893131    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.893235 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.893150    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbb151f9-f859-4faf-a830-9da6730837c6-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.894564 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.894532    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.894925 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.894902    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbb151f9-f859-4faf-a830-9da6730837c6-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.915813 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.915746    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbb151f9-f859-4faf-a830-9da6730837c6-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.916179 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.916156    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbnf5\" (UniqueName: \"kubernetes.io/projected/dbb151f9-f859-4faf-a830-9da6730837c6-kube-api-access-xbnf5\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.971185 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.971150    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:02.989260 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.989219    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"]
Oct 31 00:40:02.994167 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:02.994136    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.015195 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.015149    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0d8bae-openshift-default-dockercfg-bvb9r\""
Oct 31 00:40:03.018241 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.018209    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"]
Oct 31 00:40:03.094591 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.094554    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.094789 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.094606    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6r7s\" (UniqueName: \"kubernetes.io/projected/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-kube-api-access-m6r7s\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.094839 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.094782    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.094839 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.094825    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.094920 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.094850    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.095006 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.094968    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.095129 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.095114    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.095182 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.095164    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.095268 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.095251    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.196784 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.196660    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.196784 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.196755    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.196784 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.196781    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.197082 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.196830    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.197082 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.196862    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6r7s\" (UniqueName: \"kubernetes.io/projected/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-kube-api-access-m6r7s\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.197082 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.196892    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.197082 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.196994    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.197082 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.197047    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.197082 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.197051    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.197387 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.197087    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.197387 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.197135    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.197387 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.197219    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.197387 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.197307    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.197756 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.197734    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.199213 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.199191    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.199546 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.199529    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.220728 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.220695    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.220877 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.220781    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6r7s\" (UniqueName: \"kubernetes.io/projected/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-kube-api-access-m6r7s\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.303631 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:40:03.303603    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb151f9_f859_4faf_a830_9da6730837c6.slice/crio-06d0d477b5f29b5621123fef99b2a122f7b5828381a2ff0f9d6a38589ae985ab WatchSource:0}: Error finding container 06d0d477b5f29b5621123fef99b2a122f7b5828381a2ff0f9d6a38589ae985ab: Status 404 returned error can't find the container with id 06d0d477b5f29b5621123fef99b2a122f7b5828381a2ff0f9d6a38589ae985ab
Oct 31 00:40:03.304041 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.304015    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"]
Oct 31 00:40:03.306427 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.306395    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:40:03.306494 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.306460    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:40:03.306494 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.306489    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:40:03.315774 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.315743    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:03.628952 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.628921    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"]
Oct 31 00:40:03.630104 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:40:03.630062    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a4ff5e_3ec9_45ce_88c2_a9b6308a821e.slice/crio-ee2a1c7d5700750f28cfa7e15f9b8ba78329bc5798934095e20e11ae3aba4392 WatchSource:0}: Error finding container ee2a1c7d5700750f28cfa7e15f9b8ba78329bc5798934095e20e11ae3aba4392: Status 404 returned error can't find the container with id ee2a1c7d5700750f28cfa7e15f9b8ba78329bc5798934095e20e11ae3aba4392
Oct 31 00:40:03.633161 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.633125    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:40:03.633249 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.633198    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:40:03.633249 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.633228    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:40:03.759764 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.759724    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd" event={"ID":"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e","Type":"ContainerStarted","Data":"6ce9aee27c6c3412514e9a07f4333228e5cb3a960b110d3a1cc34ce89476d265"}
Oct 31 00:40:03.760268 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.759794    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd" event={"ID":"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e","Type":"ContainerStarted","Data":"ee2a1c7d5700750f28cfa7e15f9b8ba78329bc5798934095e20e11ae3aba4392"}
Oct 31 00:40:03.761663 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.761633    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5" event={"ID":"dbb151f9-f859-4faf-a830-9da6730837c6","Type":"ContainerStarted","Data":"074e6cb557d474457106328e29e15a36cadf922763e97e8f81bbec29f7164cc0"}
Oct 31 00:40:03.761834 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.761793    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5" event={"ID":"dbb151f9-f859-4faf-a830-9da6730837c6","Type":"ContainerStarted","Data":"06d0d477b5f29b5621123fef99b2a122f7b5828381a2ff0f9d6a38589ae985ab"}
Oct 31 00:40:03.826381 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.826301    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd" podStartSLOduration=1.826275823 podStartE2EDuration="1.826275823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:03.810159115 +0000 UTC m=+1264.032194682" watchObservedRunningTime="2026-10-31 00:40:03.826275823 +0000 UTC m=+1264.048311382"
Oct 31 00:40:03.971760 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:03.971652    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:04.320993 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:04.320907    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:04.321357 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:04.321333    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 31 00:40:04.321561 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:04.321536    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd" podUID="49a4ff5e-3ec9-45ce-88c2-a9b6308a821e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 31 00:40:04.972763 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:04.972726    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 31 00:40:04.973166 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:04.972779    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5" podUID="dbb151f9-f859-4faf-a830-9da6730837c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 31 00:40:05.324761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:05.321090    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 31 00:40:05.324761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:05.321157    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd" podUID="49a4ff5e-3ec9-45ce-88c2-a9b6308a821e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 31 00:40:05.972884 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:05.972846    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 31 00:40:05.973418 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:05.972915    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5" podUID="dbb151f9-f859-4faf-a830-9da6730837c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 31 00:40:06.316243 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:06.316164    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 31 00:40:06.316457 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:06.316435    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd" podUID="49a4ff5e-3ec9-45ce-88c2-a9b6308a821e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 31 00:40:06.972236 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:06.972197    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 31 00:40:06.972423 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:06.972252    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5" podUID="dbb151f9-f859-4faf-a830-9da6730837c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 31 00:40:07.316580 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:07.316493    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 31 00:40:07.316580 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:07.316545    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd" podUID="49a4ff5e-3ec9-45ce-88c2-a9b6308a821e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 31 00:40:07.972241 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:07.972202    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 31 00:40:07.972430 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:07.972257    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5" podUID="dbb151f9-f859-4faf-a830-9da6730837c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 31 00:40:08.321177 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:08.321080    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 31 00:40:08.321875 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:08.321841    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd" podUID="49a4ff5e-3ec9-45ce-88c2-a9b6308a821e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 31 00:40:08.972020 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:08.971981    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 31 00:40:08.972210 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:08.972032    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5" podUID="dbb151f9-f859-4faf-a830-9da6730837c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 31 00:40:09.317004 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:09.316913    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 31 00:40:09.317248 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:09.317222    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd" podUID="49a4ff5e-3ec9-45ce-88c2-a9b6308a821e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 31 00:40:10.270139 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:10.270102    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:10.271065 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:10.271026    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:10.271614 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:10.271593    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:40:10.316094 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:10.316061    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 31 00:40:10.316299 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:10.316120    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd" podUID="49a4ff5e-3ec9-45ce-88c2-a9b6308a821e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 31 00:40:10.334900 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:10.334831    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5" podStartSLOduration=8.334812006 podStartE2EDuration="8.334812006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:03.863105194 +0000 UTC m=+1264.085140737" watchObservedRunningTime="2026-10-31 00:40:10.334812006 +0000 UTC m=+1270.556847549"
Oct 31 00:40:11.321154 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:11.321114    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:11.321639 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:11.321481    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:11.322339 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:11.322323    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:40:15.292545 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:40:15.292509    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:40:45.309769 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:40:45.309723    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:40:47.653855 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.653822    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"]
Oct 31 00:40:47.658808 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.658790    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.703870 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.703829    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"]
Oct 31 00:40:47.851271 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.851234    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.851470 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.851309    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.851470 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.851361    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a45ba96-dddd-43f5-9e50-0b08194799f8-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.851470 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.851382    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.851629 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.851518    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.851629 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.851568    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.851629 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.851618    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtg4p\" (UniqueName: \"kubernetes.io/projected/2a45ba96-dddd-43f5-9e50-0b08194799f8-kube-api-access-dtg4p\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.851755 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.851643    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.851755 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.851728    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.952882 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.952806    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.952882 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.952874    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a45ba96-dddd-43f5-9e50-0b08194799f8-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.953111 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.952909    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.953111 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.952966    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.953111 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.953003    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.953111 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.953031    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtg4p\" (UniqueName: \"kubernetes.io/projected/2a45ba96-dddd-43f5-9e50-0b08194799f8-kube-api-access-dtg4p\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.953111 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.953055    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.953111 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.953080    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.953431 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.953153    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.953431 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.953387    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.953670 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.953642    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.953820 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.953713    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.953894 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.953867    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.953944 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.953884    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a45ba96-dddd-43f5-9e50-0b08194799f8-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.955445 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.955427    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.955606 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.955587    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.976706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.976665    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.976833 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.976815    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtg4p\" (UniqueName: \"kubernetes.io/projected/2a45ba96-dddd-43f5-9e50-0b08194799f8-kube-api-access-dtg4p\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:47.976982 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:47.976970    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:48.299214 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:48.299133    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"]
Oct 31 00:40:48.304611 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:40:48.304585    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a45ba96_dddd_43f5_9e50_0b08194799f8.slice/crio-54d4e4e74317d54fad5fdc84f056b0d0218492e86b4867b3b530952fc2153ac7 WatchSource:0}: Error finding container 54d4e4e74317d54fad5fdc84f056b0d0218492e86b4867b3b530952fc2153ac7: Status 404 returned error can't find the container with id 54d4e4e74317d54fad5fdc84f056b0d0218492e86b4867b3b530952fc2153ac7
Oct 31 00:40:48.307088 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:48.307051    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:40:48.307199 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:48.307135    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:40:48.307199 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:48.307177    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:40:49.149082 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:49.149046    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh" event={"ID":"2a45ba96-dddd-43f5-9e50-0b08194799f8","Type":"ContainerStarted","Data":"5637bf6d1b2b20701bc575f8f659153d51766c4bef7227af3fa93210f9101723"}
Oct 31 00:40:49.149456 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:49.149099    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh" event={"ID":"2a45ba96-dddd-43f5-9e50-0b08194799f8","Type":"ContainerStarted","Data":"54d4e4e74317d54fad5fdc84f056b0d0218492e86b4867b3b530952fc2153ac7"}
Oct 31 00:40:49.241733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:49.241614    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh" podStartSLOduration=2.241594702 podStartE2EDuration="2.241594702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:49.206387164 +0000 UTC m=+1309.428422731" watchObservedRunningTime="2026-10-31 00:40:49.241594702 +0000 UTC m=+1309.463630244"
Oct 31 00:40:49.977152 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:49.977114    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:49.978510 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:49.978486    4327 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 31 00:40:49.978577 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:49.978533    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh" podUID="2a45ba96-dddd-43f5-9e50-0b08194799f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 31 00:40:50.977536 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:50.977502    4327 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 31 00:40:50.977979 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:50.977547    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh" podUID="2a45ba96-dddd-43f5-9e50-0b08194799f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 31 00:40:51.977483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:51.977450    4327 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 31 00:40:51.977725 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:51.977507    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh" podUID="2a45ba96-dddd-43f5-9e50-0b08194799f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 31 00:40:52.978046 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:52.978008    4327 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 31 00:40:52.978557 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:52.978088    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh" podUID="2a45ba96-dddd-43f5-9e50-0b08194799f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 31 00:40:53.984798 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:53.984758    4327 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 31 00:40:53.985429 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:53.985395    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh" podUID="2a45ba96-dddd-43f5-9e50-0b08194799f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 31 00:40:55.191607 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:55.191572    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:55.192099 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:55.192050    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:40:55.193198 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:40:55.193177    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 00:41:02.948456 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:02.948420    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"]
Oct 31 00:41:02.953844 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:02.953819    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:02.973150 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:02.973121    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-be301b-openshift-default-dockercfg-4bp8d\""
Oct 31 00:41:02.988016 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:02.987992    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"]
Oct 31 00:41:03.027409 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.027378    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.027595 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.027419    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.027595 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.027568    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.027757 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.027671    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.027757 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.027738    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqspt\" (UniqueName: \"kubernetes.io/projected/be256a75-3d28-43d7-9b5d-56bf5498f0d1-kube-api-access-qqspt\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.027887 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.027766    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.027944 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.027880    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.027944 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.027913    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.028025 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.027992    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.129363 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.129327    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.129363 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.129366    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.129637 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.129401    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.129637 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.129436    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.129637 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.129460    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.129637 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.129544    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.129637 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.129606    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.129937 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.129645    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqspt\" (UniqueName: \"kubernetes.io/projected/be256a75-3d28-43d7-9b5d-56bf5498f0d1-kube-api-access-qqspt\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.129937 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.129770    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.129937 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.129857    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.130085 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.129949    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.130085 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.130002    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.130085 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.130013    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.130365 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.130348    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.131987 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.131961    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.132197 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.132179    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.178095 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.178059    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.188611 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.188579    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqspt\" (UniqueName: \"kubernetes.io/projected/be256a75-3d28-43d7-9b5d-56bf5498f0d1-kube-api-access-qqspt\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.266749 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.266644    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:03.584451 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.584422    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"]
Oct 31 00:41:03.586033 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:41:03.586001    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe256a75_3d28_43d7_9b5d_56bf5498f0d1.slice/crio-6b69effa606f03e7f6580f6729a95b82ace24083a89a738192f5a9b436bfe60d WatchSource:0}: Error finding container 6b69effa606f03e7f6580f6729a95b82ace24083a89a738192f5a9b436bfe60d: Status 404 returned error can't find the container with id 6b69effa606f03e7f6580f6729a95b82ace24083a89a738192f5a9b436bfe60d
Oct 31 00:41:03.588566 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.588535    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:41:03.588642 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.588598    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:41:03.588642 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:03.588635    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:41:04.383899 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:04.383866    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97" event={"ID":"be256a75-3d28-43d7-9b5d-56bf5498f0d1","Type":"ContainerStarted","Data":"8ae1dd10b9a8f5cb027e10bf10826d754e8bc9dd6bff45b64e005a769c8aa58f"}
Oct 31 00:41:04.384358 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:04.383921    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97" event={"ID":"be256a75-3d28-43d7-9b5d-56bf5498f0d1","Type":"ContainerStarted","Data":"6b69effa606f03e7f6580f6729a95b82ace24083a89a738192f5a9b436bfe60d"}
Oct 31 00:41:04.432179 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:04.432104    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97" podStartSLOduration=2.432090257 podStartE2EDuration="2.432090257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:04.428997509 +0000 UTC m=+1324.651033061" watchObservedRunningTime="2026-10-31 00:41:04.432090257 +0000 UTC m=+1324.654125795"
Oct 31 00:41:05.267402 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:05.267351    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:05.269195 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:05.269168    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 00:41:05.269313 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:05.269219    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97" podUID="be256a75-3d28-43d7-9b5d-56bf5498f0d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 00:41:06.267708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:06.267658    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 00:41:06.268115 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:06.267739    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97" podUID="be256a75-3d28-43d7-9b5d-56bf5498f0d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 00:41:07.267611 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:07.267575    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 00:41:07.267824 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:07.267633    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97" podUID="be256a75-3d28-43d7-9b5d-56bf5498f0d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 00:41:08.267939 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:08.267905    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 00:41:08.268357 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:08.267959    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97" podUID="be256a75-3d28-43d7-9b5d-56bf5498f0d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 00:41:09.267637 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:09.267600    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 00:41:09.267967 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:09.267922    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97" podUID="be256a75-3d28-43d7-9b5d-56bf5498f0d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 00:41:10.275708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:10.272146    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 00:41:10.275708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:10.272207    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97" podUID="be256a75-3d28-43d7-9b5d-56bf5498f0d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 00:41:11.267100 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:11.267054    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 00:41:11.267283 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:11.267130    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97" podUID="be256a75-3d28-43d7-9b5d-56bf5498f0d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 00:41:12.190438 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.190397    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"]
Oct 31 00:41:12.195634 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.195609    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.217765 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.217730    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"]
Oct 31 00:41:12.272701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.272655    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:12.273254 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.273235    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:12.274113 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.274090    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:41:12.317322 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.317283    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/738f0544-ef43-4c6c-a9ec-a20a362c482d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.317322 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.317318    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/738f0544-ef43-4c6c-a9ec-a20a362c482d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.317547 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.317358    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpx56\" (UniqueName: \"kubernetes.io/projected/738f0544-ef43-4c6c-a9ec-a20a362c482d-kube-api-access-hpx56\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.317547 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.317531    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/738f0544-ef43-4c6c-a9ec-a20a362c482d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.317622 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.317572    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/738f0544-ef43-4c6c-a9ec-a20a362c482d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.317664 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.317639    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/738f0544-ef43-4c6c-a9ec-a20a362c482d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.317853 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.317836    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/738f0544-ef43-4c6c-a9ec-a20a362c482d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.317929 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.317917    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/738f0544-ef43-4c6c-a9ec-a20a362c482d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.318090 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.318066    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/738f0544-ef43-4c6c-a9ec-a20a362c482d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.419736 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.419699    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/738f0544-ef43-4c6c-a9ec-a20a362c482d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.419946 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.419759    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/738f0544-ef43-4c6c-a9ec-a20a362c482d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.419946 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.419807    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/738f0544-ef43-4c6c-a9ec-a20a362c482d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.419946 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.419843    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/738f0544-ef43-4c6c-a9ec-a20a362c482d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.419946 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.419899    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpx56\" (UniqueName: \"kubernetes.io/projected/738f0544-ef43-4c6c-a9ec-a20a362c482d-kube-api-access-hpx56\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.419946 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.419945    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/738f0544-ef43-4c6c-a9ec-a20a362c482d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.420272 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.419971    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/738f0544-ef43-4c6c-a9ec-a20a362c482d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.420272 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.420004    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/738f0544-ef43-4c6c-a9ec-a20a362c482d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.420272 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.420099    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/738f0544-ef43-4c6c-a9ec-a20a362c482d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.420422 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.420394    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/738f0544-ef43-4c6c-a9ec-a20a362c482d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.420559 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.420534    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/738f0544-ef43-4c6c-a9ec-a20a362c482d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.420640 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.420589    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/738f0544-ef43-4c6c-a9ec-a20a362c482d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.420732 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.420671    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/738f0544-ef43-4c6c-a9ec-a20a362c482d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.421103 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.421080    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/738f0544-ef43-4c6c-a9ec-a20a362c482d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.422487 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.422461    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/738f0544-ef43-4c6c-a9ec-a20a362c482d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.422606 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.422536    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/738f0544-ef43-4c6c-a9ec-a20a362c482d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.443720 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.443624    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpx56\" (UniqueName: \"kubernetes.io/projected/738f0544-ef43-4c6c-a9ec-a20a362c482d-kube-api-access-hpx56\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.443720 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.443624    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/738f0544-ef43-4c6c-a9ec-a20a362c482d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-mbzdk\" (UID: \"738f0544-ef43-4c6c-a9ec-a20a362c482d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.510341 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.510304    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:12.831795 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.831766    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"]
Oct 31 00:41:12.833957 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:41:12.833925    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod738f0544_ef43_4c6c_a9ec_a20a362c482d.slice/crio-1638c41cf27ae60fbba35dd605ec7d69f8ddf9c7a00f2c078075ee43dbe8a6ae WatchSource:0}: Error finding container 1638c41cf27ae60fbba35dd605ec7d69f8ddf9c7a00f2c078075ee43dbe8a6ae: Status 404 returned error can't find the container with id 1638c41cf27ae60fbba35dd605ec7d69f8ddf9c7a00f2c078075ee43dbe8a6ae
Oct 31 00:41:12.836379 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.836345    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:41:12.836447 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.836416    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:41:12.836447 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:12.836442    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:41:13.482865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:13.482822    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk" event={"ID":"738f0544-ef43-4c6c-a9ec-a20a362c482d","Type":"ContainerStarted","Data":"c86dec3bafb8c7dcd5b97dc431e942ff5639bca9e8595248e371d46d22bc2313"}
Oct 31 00:41:13.483375 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:13.482879    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk" event={"ID":"738f0544-ef43-4c6c-a9ec-a20a362c482d","Type":"ContainerStarted","Data":"1638c41cf27ae60fbba35dd605ec7d69f8ddf9c7a00f2c078075ee43dbe8a6ae"}
Oct 31 00:41:13.511304 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:13.511268    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:13.514749 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:13.513205    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mbzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 00:41:13.514749 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:13.513268    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk" podUID="738f0544-ef43-4c6c-a9ec-a20a362c482d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 00:41:13.551112 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:13.551042    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk" podStartSLOduration=1.551023679 podStartE2EDuration="1.551023679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:13.541316703 +0000 UTC m=+1333.763352246" watchObservedRunningTime="2026-10-31 00:41:13.551023679 +0000 UTC m=+1333.773059226"
Oct 31 00:41:14.510893 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:14.510859    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mbzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 00:41:14.511301 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:14.510922    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk" podUID="738f0544-ef43-4c6c-a9ec-a20a362c482d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 00:41:15.292434 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:41:15.292395    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:41:15.511493 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:15.511455    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mbzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 00:41:15.511888 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:15.511529    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk" podUID="738f0544-ef43-4c6c-a9ec-a20a362c482d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 00:41:16.511117 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:16.511082    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mbzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 00:41:16.511331 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:16.511156    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk" podUID="738f0544-ef43-4c6c-a9ec-a20a362c482d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 00:41:17.511453 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:17.511413    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mbzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 00:41:17.512002 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:17.511481    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk" podUID="738f0544-ef43-4c6c-a9ec-a20a362c482d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 00:41:17.877355 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:17.877312    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"]
Oct 31 00:41:17.884414 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:17.884375    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:17.906744 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:17.906715    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"]
Oct 31 00:41:18.005867 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.005827    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02dc2389-ff84-4de4-a932-8be187b761e8-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.005867 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.005881    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02dc2389-ff84-4de4-a932-8be187b761e8-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.006111 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.006028    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.006153 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.006132    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.006206 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.006193    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.006353 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.006326    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.006441 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.006371    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02dc2389-ff84-4de4-a932-8be187b761e8-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.006548 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.006533    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.006603 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.006587    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhm6r\" (UniqueName: \"kubernetes.io/projected/02dc2389-ff84-4de4-a932-8be187b761e8-kube-api-access-zhm6r\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.108219 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.108182    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.108402 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.108235    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.108402 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.108284    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.108402 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.108315    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.108402 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.108342    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02dc2389-ff84-4de4-a932-8be187b761e8-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.108614 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.108461    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.108614 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.108498    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhm6r\" (UniqueName: \"kubernetes.io/projected/02dc2389-ff84-4de4-a932-8be187b761e8-kube-api-access-zhm6r\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.108614 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.108543    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02dc2389-ff84-4de4-a932-8be187b761e8-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.108614 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.108573    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02dc2389-ff84-4de4-a932-8be187b761e8-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.109407 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.109382    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.110023 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.109997    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.110700 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.110660    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02dc2389-ff84-4de4-a932-8be187b761e8-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.110841 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.110822    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.110901 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.110852    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.111463 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.111443    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02dc2389-ff84-4de4-a932-8be187b761e8-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.112949 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.112930    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.147084 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.146974    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02dc2389-ff84-4de4-a932-8be187b761e8-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.148110 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.148082    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhm6r\" (UniqueName: \"kubernetes.io/projected/02dc2389-ff84-4de4-a932-8be187b761e8-kube-api-access-zhm6r\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.207274 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.207242    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:18.216036 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.216008    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"]
Oct 31 00:41:18.245072 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.244965    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.246622 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.246586    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"]
Oct 31 00:41:18.313227 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.312884    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.313227 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.312954    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.313227 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.313047    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.313227 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.313091    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1ceaa6-3845-4dc6-8455-974804695d4c-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.313227 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.313134    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2k8l\" (UniqueName: \"kubernetes.io/projected/ce1ceaa6-3845-4dc6-8455-974804695d4c-kube-api-access-k2k8l\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.313227 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.313160    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.313694 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.313248    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.313694 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.313290    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.313694 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.313363    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.414630 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.414541    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.414630 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.414600    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.414630 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.414625    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.414951 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.414655    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1ceaa6-3845-4dc6-8455-974804695d4c-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.414951 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.414814    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2k8l\" (UniqueName: \"kubernetes.io/projected/ce1ceaa6-3845-4dc6-8455-974804695d4c-kube-api-access-k2k8l\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.414951 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.414863    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.414951 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.414947    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.415165 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.414988    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.415165 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.415056    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.415165 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.415098    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.415360 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.415336    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.415424 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.415051    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.415482 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.415435    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.415538 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.415515    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1ceaa6-3845-4dc6-8455-974804695d4c-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.443745 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.442306    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.443745 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.442549    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.445575 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.445197    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.445575 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.445526    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2k8l\" (UniqueName: \"kubernetes.io/projected/ce1ceaa6-3845-4dc6-8455-974804695d4c-kube-api-access-k2k8l\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.510905 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.510867    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mbzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 00:41:18.511087 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.510917    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk" podUID="738f0544-ef43-4c6c-a9ec-a20a362c482d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 00:41:18.577324 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.577285    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"]
Oct 31 00:41:18.579643 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:41:18.579617    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02dc2389_ff84_4de4_a932_8be187b761e8.slice/crio-e43a1490fcfbf86a6784cdeeaa269743efc56f57ac44dabc0fba4f340da251b0 WatchSource:0}: Error finding container e43a1490fcfbf86a6784cdeeaa269743efc56f57ac44dabc0fba4f340da251b0: Status 404 returned error can't find the container with id e43a1490fcfbf86a6784cdeeaa269743efc56f57ac44dabc0fba4f340da251b0
Oct 31 00:41:18.582180 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.582143    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:41:18.582297 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.582252    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:41:18.582362 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.582294    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:41:18.599869 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.599615    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:18.942075 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.942049    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"]
Oct 31 00:41:18.944210 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:41:18.944177    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1ceaa6_3845_4dc6_8455_974804695d4c.slice/crio-a5b906315bf935d01500b353442309d1fc456fec7765cb9806513695db392c7f WatchSource:0}: Error finding container a5b906315bf935d01500b353442309d1fc456fec7765cb9806513695db392c7f: Status 404 returned error can't find the container with id a5b906315bf935d01500b353442309d1fc456fec7765cb9806513695db392c7f
Oct 31 00:41:18.946819 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.946785    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:41:18.946918 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.946871    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:41:18.946987 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:18.946921    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:41:19.511452 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:19.511414    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mbzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 00:41:19.511650 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:19.511472    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk" podUID="738f0544-ef43-4c6c-a9ec-a20a362c482d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 00:41:19.532110 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:19.532072    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2" event={"ID":"ce1ceaa6-3845-4dc6-8455-974804695d4c","Type":"ContainerStarted","Data":"b9062b534627de3b117cabddc170a59309e3cd8e61152f5b6e47834a3e6cb9da"}
Oct 31 00:41:19.532373 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:19.532355    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2" event={"ID":"ce1ceaa6-3845-4dc6-8455-974804695d4c","Type":"ContainerStarted","Data":"a5b906315bf935d01500b353442309d1fc456fec7765cb9806513695db392c7f"}
Oct 31 00:41:19.551320 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:19.550509    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx" event={"ID":"02dc2389-ff84-4de4-a932-8be187b761e8","Type":"ContainerStarted","Data":"14b697f9593302afa5dffa82eeaa2cf945b4945a910d8c1a5a8c0379e8d890d9"}
Oct 31 00:41:19.551320 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:19.550588    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx" event={"ID":"02dc2389-ff84-4de4-a932-8be187b761e8","Type":"ContainerStarted","Data":"e43a1490fcfbf86a6784cdeeaa269743efc56f57ac44dabc0fba4f340da251b0"}
Oct 31 00:41:19.606062 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:19.603785    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:19.628702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:19.622375    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2" podStartSLOduration=1.622355236 podStartE2EDuration="1.622355236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:19.621153929 +0000 UTC m=+1339.843189490" watchObservedRunningTime="2026-10-31 00:41:19.622355236 +0000 UTC m=+1339.844390779"
Oct 31 00:41:20.207502 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:20.207451    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:20.208961 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:20.208934    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 00:41:20.209180 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:20.209157    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx" podUID="02dc2389-ff84-4de4-a932-8be187b761e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 00:41:20.510965 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:20.510872    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mbzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 00:41:20.510965 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:20.510932    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk" podUID="738f0544-ef43-4c6c-a9ec-a20a362c482d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 00:41:20.601294 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:20.601249    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 00:41:20.601472 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:20.601328    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2" podUID="ce1ceaa6-3845-4dc6-8455-974804695d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 00:41:21.209701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:21.209330    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 00:41:21.209701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:21.209411    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx" podUID="02dc2389-ff84-4de4-a932-8be187b761e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 00:41:21.600151 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:21.600127    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 00:41:21.600359 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:21.600341    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2" podUID="ce1ceaa6-3845-4dc6-8455-974804695d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 00:41:22.102516 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:22.102241    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:22.103213 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:22.103192    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:22.104193 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:22.104168    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk"
Oct 31 00:41:22.171707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:22.171511    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx" podStartSLOduration=5.171491106 podStartE2EDuration="5.171491106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:19.683451817 +0000 UTC m=+1339.905487360" watchObservedRunningTime="2026-10-31 00:41:22.171491106 +0000 UTC m=+1342.393526649"
Oct 31 00:41:22.208172 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:22.208131    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 00:41:22.208373 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:22.208203    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx" podUID="02dc2389-ff84-4de4-a932-8be187b761e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 00:41:22.600146 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:22.600110    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 00:41:22.600619 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:22.600163    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2" podUID="ce1ceaa6-3845-4dc6-8455-974804695d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 00:41:23.218165 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:23.218114    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 00:41:23.218506 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:23.218471    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx" podUID="02dc2389-ff84-4de4-a932-8be187b761e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 00:41:23.600986 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:23.600959    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 00:41:23.601380 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:23.601020    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2" podUID="ce1ceaa6-3845-4dc6-8455-974804695d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 00:41:24.208150 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:24.208092    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 00:41:24.208330 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:24.208171    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx" podUID="02dc2389-ff84-4de4-a932-8be187b761e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 00:41:24.600726 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:24.600672    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 00:41:24.600929 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:24.600746    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2" podUID="ce1ceaa6-3845-4dc6-8455-974804695d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 00:41:25.207697 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:25.207651    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 00:41:25.208100 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:25.207742    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx" podUID="02dc2389-ff84-4de4-a932-8be187b761e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 00:41:25.600935 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:25.600895    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 00:41:25.601143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:25.600966    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2" podUID="ce1ceaa6-3845-4dc6-8455-974804695d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 00:41:26.664258 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:26.664223    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:26.666132 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:26.666108    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:26.676276 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:26.676246    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:41:27.273453 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:27.273368    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:41:27.273750 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:27.273698    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx" podUID="02dc2389-ff84-4de4-a932-8be187b761e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:41:28.043704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:28.043514    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:28.063701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:28.045266    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:28.301986 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:28.301898    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:41:32.523771 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.523738    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"]
Oct 31 00:41:32.528073 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.528056    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.570415 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.570381    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"]
Oct 31 00:41:32.604711 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.604666    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"]
Oct 31 00:41:32.613944 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.613914    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.642715 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.642664    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"]
Oct 31 00:41:32.671609 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.671576    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.671609 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.671612    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.671838 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.671726    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.671838 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.671774    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.671838 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.671831    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.671938 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.671864    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.671973 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.671942    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.671973 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.671959    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrswv\" (UniqueName: \"kubernetes.io/projected/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-kube-api-access-lrswv\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.672036 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.671977    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.773215 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.773180    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.773417 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.773230    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8ft9\" (UniqueName: \"kubernetes.io/projected/9700ab24-a29a-4051-83c7-fc80e83c8ee8-kube-api-access-q8ft9\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.773417 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.773362    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.773417 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.773407    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.773593 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.773437    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.773593 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.773518    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.773593 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.773565    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.773753 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.773653    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.773828 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.773760    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.773828 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.773786    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.773930 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.773836    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.773930 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.773862    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrswv\" (UniqueName: \"kubernetes.io/projected/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-kube-api-access-lrswv\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.773930 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.773890    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.773930 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.773925    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.774195 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.773960    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.774195 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.773983    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.774195 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.774008    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.774195 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.774057    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.774195 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.774111    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.774195 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.774154    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.774457 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.774214    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.774457 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.774382    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.774659 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.774635    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.775851 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.775816    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.775985 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.775966    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.797640 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.797616    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.797823 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.797798    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrswv\" (UniqueName: \"kubernetes.io/projected/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-kube-api-access-lrswv\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.842715 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.842640    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:32.875755 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.875725    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.875929 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.875853    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.875929 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.875880    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.875929 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.875910    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.876475 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.876087    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.876475 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.876173    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.876475 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.876215    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.876475 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.876244    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.876475 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.876258    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8ft9\" (UniqueName: \"kubernetes.io/projected/9700ab24-a29a-4051-83c7-fc80e83c8ee8-kube-api-access-q8ft9\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.876475 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.876321    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.876862 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.876776    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.876862 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.876784    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.876862 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.876838    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.877027 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.876982    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.878215 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.878186    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.878650 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.878632    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.906235 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.903652    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8ft9\" (UniqueName: \"kubernetes.io/projected/9700ab24-a29a-4051-83c7-fc80e83c8ee8-kube-api-access-q8ft9\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.906235 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.905186    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:32.936140 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:32.935317    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:33.168366 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:33.168333    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"]
Oct 31 00:41:33.169237 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:41:33.169206    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c6a0bf_623b_41e0_9c60_4a3fa55e8770.slice/crio-20e2467129d098feeb82df5403873abf9922ea47faf4d70a77de3db33ab59df8 WatchSource:0}: Error finding container 20e2467129d098feeb82df5403873abf9922ea47faf4d70a77de3db33ab59df8: Status 404 returned error can't find the container with id 20e2467129d098feeb82df5403873abf9922ea47faf4d70a77de3db33ab59df8
Oct 31 00:41:33.171714 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:33.171665    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:41:33.171771 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:33.171744    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:41:33.171808 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:33.171773    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:41:33.304398 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:33.304372    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"]
Oct 31 00:41:33.307045 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:41:33.306876    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9700ab24_a29a_4051_83c7_fc80e83c8ee8.slice/crio-e975d1280d7a35e2c77ff7713f04f44aec4a076c2e1f2fca39dbfef9e1199924 WatchSource:0}: Error finding container e975d1280d7a35e2c77ff7713f04f44aec4a076c2e1f2fca39dbfef9e1199924: Status 404 returned error can't find the container with id e975d1280d7a35e2c77ff7713f04f44aec4a076c2e1f2fca39dbfef9e1199924
Oct 31 00:41:33.311339 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:33.311297    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:41:33.311455 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:33.311386    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:41:33.311455 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:33.311428    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:41:33.710050 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:33.710011    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz" event={"ID":"9700ab24-a29a-4051-83c7-fc80e83c8ee8","Type":"ContainerStarted","Data":"a66e34b4b1c659c0dd576a9b2da81dc9bd363591e70e7c22c2e8e8b989847f36"}
Oct 31 00:41:33.710534 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:33.710075    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz" event={"ID":"9700ab24-a29a-4051-83c7-fc80e83c8ee8","Type":"ContainerStarted","Data":"e975d1280d7a35e2c77ff7713f04f44aec4a076c2e1f2fca39dbfef9e1199924"}
Oct 31 00:41:33.711917 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:33.711881    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7" event={"ID":"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770","Type":"ContainerStarted","Data":"9a185e9c58176b62d3a4e9195aa9e1607687144d3bbc1d0ba299ace77be5fb96"}
Oct 31 00:41:33.712031 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:33.711938    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7" event={"ID":"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770","Type":"ContainerStarted","Data":"20e2467129d098feeb82df5403873abf9922ea47faf4d70a77de3db33ab59df8"}
Oct 31 00:41:33.757793 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:33.757741    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz" podStartSLOduration=1.757727539 podStartE2EDuration="1.757727539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:33.755501051 +0000 UTC m=+1353.977536595" watchObservedRunningTime="2026-10-31 00:41:33.757727539 +0000 UTC m=+1353.979763081"
Oct 31 00:41:33.801423 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:33.801356    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7" podStartSLOduration=1.801339394 podStartE2EDuration="1.801339394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:33.800604224 +0000 UTC m=+1354.022639766" watchObservedRunningTime="2026-10-31 00:41:33.801339394 +0000 UTC m=+1354.023374936"
Oct 31 00:41:33.843814 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:33.843724    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:33.938827 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:33.938789    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:33.990895 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:33.990351    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"]
Oct 31 00:41:33.995478 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:33.995453    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:41:34.011078 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:34.011053    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0d8bae\"/\"distinct-2-w4-0d8bae-kserve-self-signed-certs\""
Oct 31 00:41:34.020702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:34.017286    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"]
Oct 31 00:41:34.096717 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:34.094131    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-dshm\") pod \"distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8\" (UID: \"b6a8824b-19f2-47df-88e5-442724a36100\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:41:34.096717 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:34.094174    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-model-cache\") pod \"distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8\" (UID: \"b6a8824b-19f2-47df-88e5-442724a36100\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:41:34.096717 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:34.094207    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-home\") pod \"distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8\" (UID: \"b6a8824b-19f2-47df-88e5-442724a36100\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:41:34.096717 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:34.094222    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-tmp-dir\") pod \"distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8\" (UID: \"b6a8824b-19f2-47df-88e5-442724a36100\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:41:34.096717 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:34.094260    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a8824b-19f2-47df-88e5-442724a36100-tls-certs\") pod \"distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8\" (UID: \"b6a8824b-19f2-47df-88e5-442724a36100\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:41:34.096717 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:34.094280    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk566\" (UniqueName: \"kubernetes.io/projected/b6a8824b-19f2-47df-88e5-442724a36100-kube-api-access-pk566\") pod \"distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8\" (UID: \"b6a8824b-19f2-47df-88e5-442724a36100\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:41:34.227701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:34.227594    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-dshm\") pod \"distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8\" (UID: \"b6a8824b-19f2-47df-88e5-442724a36100\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:41:34.227701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:34.227654    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-model-cache\") pod \"distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8\" (UID: \"b6a8824b-19f2-47df-88e5-442724a36100\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:41:34.227971 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:34.227723    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-home\") pod \"distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8\" (UID: \"b6a8824b-19f2-47df-88e5-442724a36100\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:41:34.227971 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:34.227744    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-tmp-dir\") pod \"distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8\" (UID: \"b6a8824b-19f2-47df-88e5-442724a36100\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:41:34.227971 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:34.227797    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a8824b-19f2-47df-88e5-442724a36100-tls-certs\") pod \"distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8\" (UID: \"b6a8824b-19f2-47df-88e5-442724a36100\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:41:34.227971 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:34.227834    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk566\" (UniqueName: \"kubernetes.io/projected/b6a8824b-19f2-47df-88e5-442724a36100-kube-api-access-pk566\") pod \"distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8\" (UID: \"b6a8824b-19f2-47df-88e5-442724a36100\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:41:34.268270 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:34.268190    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-home\") pod \"distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8\" (UID: \"b6a8824b-19f2-47df-88e5-442724a36100\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:41:34.277867 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:34.274498    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-tmp-dir\") pod \"distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8\" (UID: \"b6a8824b-19f2-47df-88e5-442724a36100\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:41:34.277867 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:34.274757    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-model-cache\") pod \"distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8\" (UID: \"b6a8824b-19f2-47df-88e5-442724a36100\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:41:34.302705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:34.284278    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a8824b-19f2-47df-88e5-442724a36100-tls-certs\") pod \"distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8\" (UID: \"b6a8824b-19f2-47df-88e5-442724a36100\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:41:34.323705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:34.312431    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-dshm\") pod \"distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8\" (UID: \"b6a8824b-19f2-47df-88e5-442724a36100\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:41:34.371710 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:34.369438    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk566\" (UniqueName: \"kubernetes.io/projected/b6a8824b-19f2-47df-88e5-442724a36100-kube-api-access-pk566\") pod \"distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8\" (UID: \"b6a8824b-19f2-47df-88e5-442724a36100\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:41:34.613067 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:34.613034    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:41:34.868050 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:34.867954    4327 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 00:41:34.868050 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:34.868022    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7" podUID="c5c6a0bf-623b-41e0-9c60-4a3fa55e8770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 00:41:34.973177 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:34.972818    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 31 00:41:34.973177 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:34.972880    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz" podUID="9700ab24-a29a-4051-83c7-fc80e83c8ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 31 00:41:35.010097 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:35.009964    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"]
Oct 31 00:41:35.013029 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:41:35.012999    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6a8824b_19f2_47df_88e5_442724a36100.slice/crio-e0cbe8cd5cbfd2d9b956f40302ea288a3a3169708eea915e4ce63f0817053d65 WatchSource:0}: Error finding container e0cbe8cd5cbfd2d9b956f40302ea288a3a3169708eea915e4ce63f0817053d65: Status 404 returned error can't find the container with id e0cbe8cd5cbfd2d9b956f40302ea288a3a3169708eea915e4ce63f0817053d65
Oct 31 00:41:35.770232 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:35.769948    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" event={"ID":"b6a8824b-19f2-47df-88e5-442724a36100","Type":"ContainerStarted","Data":"6c40379f8bbb6dd7b412610d94d5047a961890861a904e5bcfec28aaf7411142"}
Oct 31 00:41:35.770232 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:35.770005    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" event={"ID":"b6a8824b-19f2-47df-88e5-442724a36100","Type":"ContainerStarted","Data":"e0cbe8cd5cbfd2d9b956f40302ea288a3a3169708eea915e4ce63f0817053d65"}
Oct 31 00:41:35.815790 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:35.815742    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podStartSLOduration=2.815728424 podStartE2EDuration="2.815728424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:35.813249288 +0000 UTC m=+1356.035284837" watchObservedRunningTime="2026-10-31 00:41:35.815728424 +0000 UTC m=+1356.037763965"
Oct 31 00:41:35.851559 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:35.851105    4327 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 00:41:35.851559 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:35.851169    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7" podUID="c5c6a0bf-623b-41e0-9c60-4a3fa55e8770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 00:41:35.938051 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:35.938016    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 31 00:41:35.938752 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:35.938718    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz" podUID="9700ab24-a29a-4051-83c7-fc80e83c8ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 31 00:41:36.613316 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:36.613284    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:41:36.858728 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:36.853816    4327 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 00:41:36.858728 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:36.853872    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7" podUID="c5c6a0bf-623b-41e0-9c60-4a3fa55e8770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 00:41:36.942779 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:36.941148    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 31 00:41:36.942779 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:36.941215    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz" podUID="9700ab24-a29a-4051-83c7-fc80e83c8ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 31 00:41:37.844129 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:37.844089    4327 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 00:41:37.844427 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:37.844397    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7" podUID="c5c6a0bf-623b-41e0-9c60-4a3fa55e8770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 00:41:37.936157 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:37.936116    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 31 00:41:37.936448 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:37.936419    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz" podUID="9700ab24-a29a-4051-83c7-fc80e83c8ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 31 00:41:38.849704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:38.849258    4327 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 00:41:38.849704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:38.849324    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7" podUID="c5c6a0bf-623b-41e0-9c60-4a3fa55e8770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 00:41:38.939705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:38.939128    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 31 00:41:38.939705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:38.939191    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz" podUID="9700ab24-a29a-4051-83c7-fc80e83c8ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 31 00:41:39.856742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:39.853619    4327 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 00:41:39.856742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:39.853699    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7" podUID="c5c6a0bf-623b-41e0-9c60-4a3fa55e8770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 00:41:39.939704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:39.938704    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 31 00:41:39.939704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:39.938771    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz" podUID="9700ab24-a29a-4051-83c7-fc80e83c8ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 31 00:41:41.152933 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:41.152896    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:41.153891 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:41.153852    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:41.155553 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:41.155529    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:41:41.306635 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:41.306605    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:41.306998 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:41.306973    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:41.309236 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:41.309214    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:41:44.613300 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:44.613264    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:41:44.625955 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:44.625926    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:41:44.947934 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:44.947904    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:41:45.293313 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:41:45.293216    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:41:47.297209 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.297160    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"]
Oct 31 00:41:47.301533 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.301512    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.327071 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.327034    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"]
Oct 31 00:41:47.443149 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.443115    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.443325 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.443154    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c74e25-811d-414f-b093-a907ca85631d-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.443325 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.443178    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c74e25-811d-414f-b093-a907ca85631d-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.443403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.443323    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.443465 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.443451    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.443501 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.443483    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.443534 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.443513    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c74e25-811d-414f-b093-a907ca85631d-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.443591 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.443579    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlxkt\" (UniqueName: \"kubernetes.io/projected/c9c74e25-811d-414f-b093-a907ca85631d-kube-api-access-dlxkt\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.443626 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.443603    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.544742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.544667    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.544944 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.544763    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.544944 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.544806    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c74e25-811d-414f-b093-a907ca85631d-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.544944 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.544846    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlxkt\" (UniqueName: \"kubernetes.io/projected/c9c74e25-811d-414f-b093-a907ca85631d-kube-api-access-dlxkt\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.544944 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.544871    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.544944 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.544942    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.545204 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.544967    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c74e25-811d-414f-b093-a907ca85631d-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.545204 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.544997    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c74e25-811d-414f-b093-a907ca85631d-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.545204 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.545068    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.545204 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.545088    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.545418 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.545358    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.545418 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.545410    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.545559 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.545516    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.545892 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.545868    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c74e25-811d-414f-b093-a907ca85631d-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.547320 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.547264    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.547564 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.547548    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c74e25-811d-414f-b093-a907ca85631d-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.568564 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.568534    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c74e25-811d-414f-b093-a907ca85631d-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.568716 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.568640    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlxkt\" (UniqueName: \"kubernetes.io/projected/c9c74e25-811d-414f-b093-a907ca85631d-kube-api-access-dlxkt\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.617289 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.617252    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:47.851595 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.851552    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"]
Oct 31 00:41:47.856971 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.856945    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:47.882771 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.882735    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"]
Oct 31 00:41:47.940778 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.940744    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"]
Oct 31 00:41:47.942545 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:41:47.942520    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c74e25_811d_414f_b093_a907ca85631d.slice/crio-b437fc9315590ad3f22ffc6e9681a14dc12d326119ff3b28ff6cfbf58a3ef52f WatchSource:0}: Error finding container b437fc9315590ad3f22ffc6e9681a14dc12d326119ff3b28ff6cfbf58a3ef52f: Status 404 returned error can't find the container with id b437fc9315590ad3f22ffc6e9681a14dc12d326119ff3b28ff6cfbf58a3ef52f
Oct 31 00:41:47.945431 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.945395    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:41:47.945515 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.945463    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:41:47.945515 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.945499    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:41:47.955406 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:47.955380    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8" event={"ID":"c9c74e25-811d-414f-b093-a907ca85631d","Type":"ContainerStarted","Data":"b437fc9315590ad3f22ffc6e9681a14dc12d326119ff3b28ff6cfbf58a3ef52f"}
Oct 31 00:41:48.125163 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.125060    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k6rh\" (UniqueName: \"kubernetes.io/projected/7b9657eb-33b5-4847-9ce7-39085e57f7ae-kube-api-access-5k6rh\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.125163 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.125130    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.125396 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.125208    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.125396 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.125276    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.125396 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.125348    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.125554 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.125480    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.125554 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.125519    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.125662 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.125644    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.127749 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.126848    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.228152 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.228109    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.228364 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.228158    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.228364 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.228218    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.228364 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.228265    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.228364 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.228337    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k6rh\" (UniqueName: \"kubernetes.io/projected/7b9657eb-33b5-4847-9ce7-39085e57f7ae-kube-api-access-5k6rh\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.228364 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.228363    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.228643 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.228405    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.228643 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.228444    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.228643 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.228507    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.228839 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.228726    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.228839 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.228750    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.228981 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.228852    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.229038 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.229012    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.229579 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.229545    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.230731 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.230709    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.230935 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.230914    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.252908 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.252876    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.253072 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.253053    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k6rh\" (UniqueName: \"kubernetes.io/projected/7b9657eb-33b5-4847-9ce7-39085e57f7ae-kube-api-access-5k6rh\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.473881 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.473451    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:48.846604 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.846574    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"]
Oct 31 00:41:48.848308 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:41:48.848276    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9657eb_33b5_4847_9ce7_39085e57f7ae.slice/crio-a48244fd20ac16db7a97ba010c43dea3f2f6c5d3d7a743326d6a57c18dfd52bf WatchSource:0}: Error finding container a48244fd20ac16db7a97ba010c43dea3f2f6c5d3d7a743326d6a57c18dfd52bf: Status 404 returned error can't find the container with id a48244fd20ac16db7a97ba010c43dea3f2f6c5d3d7a743326d6a57c18dfd52bf
Oct 31 00:41:48.850791 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.850754    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:41:48.850882 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.850833    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:41:48.850882 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.850865    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:41:48.962987 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.962919    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8" event={"ID":"c9c74e25-811d-414f-b093-a907ca85631d","Type":"ContainerStarted","Data":"925e9b469f952663dde1cd4de03c94b30b5bb689422e8a82e005a6d2723da3ab"}
Oct 31 00:41:48.964835 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.964764    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd" event={"ID":"7b9657eb-33b5-4847-9ce7-39085e57f7ae","Type":"ContainerStarted","Data":"dc3ade288691c2bfaa1403c7bfa95e3c3ac0ece34574b479476086504926aad5"}
Oct 31 00:41:48.964835 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:48.964821    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd" event={"ID":"7b9657eb-33b5-4847-9ce7-39085e57f7ae","Type":"ContainerStarted","Data":"a48244fd20ac16db7a97ba010c43dea3f2f6c5d3d7a743326d6a57c18dfd52bf"}
Oct 31 00:41:49.045699 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:49.040735    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8" podStartSLOduration=2.040713235 podStartE2EDuration="2.040713235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:49.030974528 +0000 UTC m=+1369.253010067" watchObservedRunningTime="2026-10-31 00:41:49.040713235 +0000 UTC m=+1369.262748777"
Oct 31 00:41:49.086823 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:49.086765    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd" podStartSLOduration=2.086749689 podStartE2EDuration="2.086749689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:49.083354674 +0000 UTC m=+1369.305390219" watchObservedRunningTime="2026-10-31 00:41:49.086749689 +0000 UTC m=+1369.308785230"
Oct 31 00:41:49.475807 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:49.475766    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:49.476608 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:49.476586    4327 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 00:41:49.476774 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:49.476633    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd" podUID="7b9657eb-33b5-4847-9ce7-39085e57f7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 00:41:49.626712 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:49.625918    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:49.647707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:49.646862    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 00:41:49.647707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:49.646960    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8" podUID="c9c74e25-811d-414f-b093-a907ca85631d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 00:41:50.474976 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:50.474943    4327 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 00:41:50.475226 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:50.475204    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd" podUID="7b9657eb-33b5-4847-9ce7-39085e57f7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 00:41:50.642703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:50.629093    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 00:41:50.642703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:50.629157    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8" podUID="c9c74e25-811d-414f-b093-a907ca85631d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 00:41:51.475287 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:51.475155    4327 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 00:41:51.475287 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:51.475204    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd" podUID="7b9657eb-33b5-4847-9ce7-39085e57f7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 00:41:51.631149 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:51.631111    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 00:41:51.631441 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:51.631392    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8" podUID="c9c74e25-811d-414f-b093-a907ca85631d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 00:41:52.475850 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:52.475804    4327 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 00:41:52.476259 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:52.475870    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd" podUID="7b9657eb-33b5-4847-9ce7-39085e57f7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 00:41:52.621068 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:52.621031    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 00:41:52.621360 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:52.621327    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8" podUID="c9c74e25-811d-414f-b093-a907ca85631d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 00:41:53.475132 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:53.475090    4327 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 00:41:53.475444 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:53.475403    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd" podUID="7b9657eb-33b5-4847-9ce7-39085e57f7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 00:41:53.625708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:53.621015    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 00:41:53.625708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:53.621076    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8" podUID="c9c74e25-811d-414f-b093-a907ca85631d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 00:41:54.475344 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:54.475306    4327 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 00:41:54.475524 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:54.475390    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd" podUID="7b9657eb-33b5-4847-9ce7-39085e57f7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 00:41:54.626106 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:54.626064    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 00:41:54.626767 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:54.626732    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8" podUID="c9c74e25-811d-414f-b093-a907ca85631d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 00:41:55.475145 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:55.475107    4327 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 00:41:55.475339 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:55.475166    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd" podUID="7b9657eb-33b5-4847-9ce7-39085e57f7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 00:41:55.714032 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:55.713998    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:55.715182 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:55.715153    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:55.733768 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:55.732007    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:41:55.939413 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:55.939365    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"]
Oct 31 00:41:55.944295 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:55.944268    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:41:55.967820 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:55.967786    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c9e377\"/\"distinct-w5-c9e377-kserve-self-signed-certs\""
Oct 31 00:41:55.971650 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:55.971616    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"]
Oct 31 00:41:56.023415 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:56.023326    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-home\") pod \"distinct-w5-c9e377-kserve-579796bcfd-dnnl5\" (UID: \"6deba2aa-764d-4325-8c5d-c09fed312bb6\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:41:56.023415 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:56.023362    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6deba2aa-764d-4325-8c5d-c09fed312bb6-tls-certs\") pod \"distinct-w5-c9e377-kserve-579796bcfd-dnnl5\" (UID: \"6deba2aa-764d-4325-8c5d-c09fed312bb6\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:41:56.023415 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:56.023387    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-model-cache\") pod \"distinct-w5-c9e377-kserve-579796bcfd-dnnl5\" (UID: \"6deba2aa-764d-4325-8c5d-c09fed312bb6\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:41:56.023415 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:56.023416    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fprx\" (UniqueName: \"kubernetes.io/projected/6deba2aa-764d-4325-8c5d-c09fed312bb6-kube-api-access-5fprx\") pod \"distinct-w5-c9e377-kserve-579796bcfd-dnnl5\" (UID: \"6deba2aa-764d-4325-8c5d-c09fed312bb6\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:41:56.023702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:56.023582    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-dshm\") pod \"distinct-w5-c9e377-kserve-579796bcfd-dnnl5\" (UID: \"6deba2aa-764d-4325-8c5d-c09fed312bb6\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:41:56.023702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:56.023636    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-tmp-dir\") pod \"distinct-w5-c9e377-kserve-579796bcfd-dnnl5\" (UID: \"6deba2aa-764d-4325-8c5d-c09fed312bb6\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:41:56.124874 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:56.124835    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-home\") pod \"distinct-w5-c9e377-kserve-579796bcfd-dnnl5\" (UID: \"6deba2aa-764d-4325-8c5d-c09fed312bb6\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:41:56.124874 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:56.124878    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6deba2aa-764d-4325-8c5d-c09fed312bb6-tls-certs\") pod \"distinct-w5-c9e377-kserve-579796bcfd-dnnl5\" (UID: \"6deba2aa-764d-4325-8c5d-c09fed312bb6\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:41:56.125080 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:56.124905    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-model-cache\") pod \"distinct-w5-c9e377-kserve-579796bcfd-dnnl5\" (UID: \"6deba2aa-764d-4325-8c5d-c09fed312bb6\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:41:56.125080 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:56.125021    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fprx\" (UniqueName: \"kubernetes.io/projected/6deba2aa-764d-4325-8c5d-c09fed312bb6-kube-api-access-5fprx\") pod \"distinct-w5-c9e377-kserve-579796bcfd-dnnl5\" (UID: \"6deba2aa-764d-4325-8c5d-c09fed312bb6\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:41:56.125159 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:56.125092    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-dshm\") pod \"distinct-w5-c9e377-kserve-579796bcfd-dnnl5\" (UID: \"6deba2aa-764d-4325-8c5d-c09fed312bb6\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:41:56.125159 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:56.125134    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-tmp-dir\") pod \"distinct-w5-c9e377-kserve-579796bcfd-dnnl5\" (UID: \"6deba2aa-764d-4325-8c5d-c09fed312bb6\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:41:56.125362 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:56.125340    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-model-cache\") pod \"distinct-w5-c9e377-kserve-579796bcfd-dnnl5\" (UID: \"6deba2aa-764d-4325-8c5d-c09fed312bb6\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:41:56.125461 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:56.125369    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-home\") pod \"distinct-w5-c9e377-kserve-579796bcfd-dnnl5\" (UID: \"6deba2aa-764d-4325-8c5d-c09fed312bb6\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:41:56.125517 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:56.125496    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-tmp-dir\") pod \"distinct-w5-c9e377-kserve-579796bcfd-dnnl5\" (UID: \"6deba2aa-764d-4325-8c5d-c09fed312bb6\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:41:56.127306 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:56.127279    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-dshm\") pod \"distinct-w5-c9e377-kserve-579796bcfd-dnnl5\" (UID: \"6deba2aa-764d-4325-8c5d-c09fed312bb6\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:41:56.127407 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:56.127387    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6deba2aa-764d-4325-8c5d-c09fed312bb6-tls-certs\") pod \"distinct-w5-c9e377-kserve-579796bcfd-dnnl5\" (UID: \"6deba2aa-764d-4325-8c5d-c09fed312bb6\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:41:56.182181 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:56.177181    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fprx\" (UniqueName: \"kubernetes.io/projected/6deba2aa-764d-4325-8c5d-c09fed312bb6-kube-api-access-5fprx\") pod \"distinct-w5-c9e377-kserve-579796bcfd-dnnl5\" (UID: \"6deba2aa-764d-4325-8c5d-c09fed312bb6\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:41:56.266987 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:56.266653    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:41:56.479661 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:56.479632    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:56.479959 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:56.479937    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:56.480928 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:56.480818    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 00:41:56.606980 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:56.604901    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"]
Oct 31 00:41:56.614863 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:41:56.614833    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6deba2aa_764d_4325_8c5d_c09fed312bb6.slice/crio-0b93fd1243ce26501cc9cf39f23277cb3e94211ef3e386fec04ea56a662d703b WatchSource:0}: Error finding container 0b93fd1243ce26501cc9cf39f23277cb3e94211ef3e386fec04ea56a662d703b: Status 404 returned error can't find the container with id 0b93fd1243ce26501cc9cf39f23277cb3e94211ef3e386fec04ea56a662d703b
Oct 31 00:41:57.107626 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:57.107591    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" event={"ID":"6deba2aa-764d-4325-8c5d-c09fed312bb6","Type":"ContainerStarted","Data":"4d06b050052daac737055df47d3866f116fc6da948fd5b2e65afd3ba2d9b17db"}
Oct 31 00:41:57.108105 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:57.107638    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" event={"ID":"6deba2aa-764d-4325-8c5d-c09fed312bb6","Type":"ContainerStarted","Data":"0b93fd1243ce26501cc9cf39f23277cb3e94211ef3e386fec04ea56a662d703b"}
Oct 31 00:41:57.153642 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:57.153591    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podStartSLOduration=2.153576535 podStartE2EDuration="2.153576535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:57.15277112 +0000 UTC m=+1377.374806665" watchObservedRunningTime="2026-10-31 00:41:57.153576535 +0000 UTC m=+1377.375612077"
Oct 31 00:41:57.271886 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:57.271807    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:41:59.990703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:41:59.990178    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:42:00.183704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:00.183061    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded"
Oct 31 00:42:06.267631 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:06.267593    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:42:06.281253 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:06.281224    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:42:06.292725 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:06.292701    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:42:10.654582 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:10.654546    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"]
Oct 31 00:42:10.660168 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:10.660142    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:42:10.676873 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:10.676810    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-671959\"/\"openshift-service-ca.crt\""
Oct 31 00:42:10.676873 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:10.676851    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-671959\"/\"kube-root-ca.crt\""
Oct 31 00:42:10.689559 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:10.689530    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-671959\"/\"default-dockercfg-m5n9g\""
Oct 31 00:42:10.689748 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:10.689731    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-671959\"/\"distinct-2-w0-671959-kserve-self-signed-certs\""
Oct 31 00:42:10.692672 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:10.692648    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"]
Oct 31 00:42:10.697295 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:10.697262    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-home\") pod \"distinct-2-w0-671959-kserve-7dccd77d97-q8wzs\" (UID: \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\") " pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:42:10.697413 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:10.697318    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f30ed7-5edf-403e-b75c-53f8b27f94bd-tls-certs\") pod \"distinct-2-w0-671959-kserve-7dccd77d97-q8wzs\" (UID: \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\") " pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:42:10.697413 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:10.697348    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-dshm\") pod \"distinct-2-w0-671959-kserve-7dccd77d97-q8wzs\" (UID: \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\") " pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:42:10.697536 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:10.697401    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-tmp-dir\") pod \"distinct-2-w0-671959-kserve-7dccd77d97-q8wzs\" (UID: \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\") " pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:42:10.697603 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:10.697585    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-model-cache\") pod \"distinct-2-w0-671959-kserve-7dccd77d97-q8wzs\" (UID: \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\") " pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:42:10.697662 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:10.697618    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7klf\" (UniqueName: \"kubernetes.io/projected/52f30ed7-5edf-403e-b75c-53f8b27f94bd-kube-api-access-w7klf\") pod \"distinct-2-w0-671959-kserve-7dccd77d97-q8wzs\" (UID: \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\") " pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:42:10.799178 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:10.799143    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7klf\" (UniqueName: \"kubernetes.io/projected/52f30ed7-5edf-403e-b75c-53f8b27f94bd-kube-api-access-w7klf\") pod \"distinct-2-w0-671959-kserve-7dccd77d97-q8wzs\" (UID: \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\") " pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:42:10.799386 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:10.799238    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-home\") pod \"distinct-2-w0-671959-kserve-7dccd77d97-q8wzs\" (UID: \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\") " pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:42:10.799386 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:10.799276    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f30ed7-5edf-403e-b75c-53f8b27f94bd-tls-certs\") pod \"distinct-2-w0-671959-kserve-7dccd77d97-q8wzs\" (UID: \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\") " pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:42:10.799386 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:10.799296    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-dshm\") pod \"distinct-2-w0-671959-kserve-7dccd77d97-q8wzs\" (UID: \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\") " pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:42:10.799560 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:10.799537    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-tmp-dir\") pod \"distinct-2-w0-671959-kserve-7dccd77d97-q8wzs\" (UID: \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\") " pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:42:10.799742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:10.799718    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-home\") pod \"distinct-2-w0-671959-kserve-7dccd77d97-q8wzs\" (UID: \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\") " pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:42:10.799852 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:10.799811    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-model-cache\") pod \"distinct-2-w0-671959-kserve-7dccd77d97-q8wzs\" (UID: \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\") " pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:42:10.799940 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:10.799917    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-tmp-dir\") pod \"distinct-2-w0-671959-kserve-7dccd77d97-q8wzs\" (UID: \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\") " pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:42:10.800167 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:10.800145    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-model-cache\") pod \"distinct-2-w0-671959-kserve-7dccd77d97-q8wzs\" (UID: \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\") " pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:42:10.801762 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:10.801738    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-dshm\") pod \"distinct-2-w0-671959-kserve-7dccd77d97-q8wzs\" (UID: \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\") " pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:42:10.802040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:10.802022    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f30ed7-5edf-403e-b75c-53f8b27f94bd-tls-certs\") pod \"distinct-2-w0-671959-kserve-7dccd77d97-q8wzs\" (UID: \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\") " pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:42:10.826954 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:10.826928    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7klf\" (UniqueName: \"kubernetes.io/projected/52f30ed7-5edf-403e-b75c-53f8b27f94bd-kube-api-access-w7klf\") pod \"distinct-2-w0-671959-kserve-7dccd77d97-q8wzs\" (UID: \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\") " pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:42:10.978753 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:10.973933    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:42:11.348557 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:11.348532    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"]
Oct 31 00:42:11.350542 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:42:11.350516    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52f30ed7_5edf_403e_b75c_53f8b27f94bd.slice/crio-c8b5d983abc2117f6da897f45e23e9c5b4bceebabdf8bf1b239fafd112aaf012 WatchSource:0}: Error finding container c8b5d983abc2117f6da897f45e23e9c5b4bceebabdf8bf1b239fafd112aaf012: Status 404 returned error can't find the container with id c8b5d983abc2117f6da897f45e23e9c5b4bceebabdf8bf1b239fafd112aaf012
Oct 31 00:42:12.318846 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:12.318806    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" event={"ID":"52f30ed7-5edf-403e-b75c-53f8b27f94bd","Type":"ContainerStarted","Data":"1d9f01421ca4002b44bf489e9ed02ef068d5ab583d9036cb075ac478a0323b48"}
Oct 31 00:42:12.319243 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:12.318857    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" event={"ID":"52f30ed7-5edf-403e-b75c-53f8b27f94bd","Type":"ContainerStarted","Data":"c8b5d983abc2117f6da897f45e23e9c5b4bceebabdf8bf1b239fafd112aaf012"}
Oct 31 00:42:12.364110 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:12.364053    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podStartSLOduration=2.364039997 podStartE2EDuration="2.364039997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:12.36084703 +0000 UTC m=+1392.582882577" watchObservedRunningTime="2026-10-31 00:42:12.364039997 +0000 UTC m=+1392.586075596"
Oct 31 00:42:12.974944 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:12.974905    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:42:15.296154 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:42:15.296111    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:42:17.703704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.702184    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"]
Oct 31 00:42:17.711272 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.711229    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:17.753078 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.752784    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"]
Oct 31 00:42:17.813789 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.808043    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"]
Oct 31 00:42:17.851115 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.851062    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32926f1-4427-4764-b86d-7294eaba9e0f-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:17.851320 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.851180    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:17.851320 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.851248    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:17.851320 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.851274    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:17.851505 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.851327    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:17.851505 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.851426    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:17.851505 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.851469    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:17.851671 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.851550    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:17.851783 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.851734    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w6vt\" (UniqueName: \"kubernetes.io/projected/f32926f1-4427-4764-b86d-7294eaba9e0f-kube-api-access-5w6vt\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:17.871899 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.871866    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"]
Oct 31 00:42:17.872278 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.872257    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:17.964984 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.964648    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:17.964984 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.964718    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:17.965237 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.965143    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:17.965237 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.965179    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:17.965237 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.965213    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll52x\" (UniqueName: \"kubernetes.io/projected/93505d53-b42a-4c07-ae52-d62ac15f1d79-kube-api-access-ll52x\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:17.965408 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.965247    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w6vt\" (UniqueName: \"kubernetes.io/projected/f32926f1-4427-4764-b86d-7294eaba9e0f-kube-api-access-5w6vt\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:17.965408 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.965277    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32926f1-4427-4764-b86d-7294eaba9e0f-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:17.965408 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.965346    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:17.965408 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.965380    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:17.965638 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.965429    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:17.965638 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.965456    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:17.965638 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.965504    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:17.965638 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.965537    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:17.965638 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.965568    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:17.965941 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.965656    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:17.965941 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.965717    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:17.965941 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.965795    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:17.965941 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.965830    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93505d53-b42a-4c07-ae52-d62ac15f1d79-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:17.967631 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.967605    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:17.968114 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.968090    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:17.969849 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.969499    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:17.969849 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.969790    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:17.970586 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.970566    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32926f1-4427-4764-b86d-7294eaba9e0f-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:17.971494 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.971472    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:17.974591 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:17.974568    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:18.043445 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.043371    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:18.072850 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.068488    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:18.072850 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.068541    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:18.072850 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.068931    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93505d53-b42a-4c07-ae52-d62ac15f1d79-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:18.072850 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.069013    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:18.072850 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.069043    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:18.072850 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.069128    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:18.072850 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.069153    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:18.072850 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.069207    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll52x\" (UniqueName: \"kubernetes.io/projected/93505d53-b42a-4c07-ae52-d62ac15f1d79-kube-api-access-ll52x\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:18.072850 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.069315    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:18.078592 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.074101    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:18.078592 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.074394    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:18.078592 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.075634    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93505d53-b42a-4c07-ae52-d62ac15f1d79-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:18.078592 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.075937    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:18.078592 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.076169    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:18.117627 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.117159    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:18.122606 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.122527    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w6vt\" (UniqueName: \"kubernetes.io/projected/f32926f1-4427-4764-b86d-7294eaba9e0f-kube-api-access-5w6vt\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-cddfp\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:18.147300 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.147189    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:18.148497 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.148390    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:18.152488 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.152443    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll52x\" (UniqueName: \"kubernetes.io/projected/93505d53-b42a-4c07-ae52-d62ac15f1d79-kube-api-access-ll52x\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:18.213284 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.213251    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:18.377429 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.377398    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:18.432436 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.432403    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"]
Oct 31 00:42:18.439065 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.439037    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.452434 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.451337    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"]
Oct 31 00:42:18.458472 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.458447    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.487790 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.487756    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"]
Oct 31 00:42:18.491782 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.491761    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"]
Oct 31 00:42:18.538080 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.537657    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"]
Oct 31 00:42:18.539558 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:42:18.539531    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93505d53_b42a_4c07_ae52_d62ac15f1d79.slice/crio-e1cb77d4331d73b3ed66491f2179c82a298e599ecfbcd7e02dac02099d719f27 WatchSource:0}: Error finding container e1cb77d4331d73b3ed66491f2179c82a298e599ecfbcd7e02dac02099d719f27: Status 404 returned error can't find the container with id e1cb77d4331d73b3ed66491f2179c82a298e599ecfbcd7e02dac02099d719f27
Oct 31 00:42:18.542508 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.542473    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:42:18.542610 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.542550    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:42:18.542674 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.542615    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:42:18.580313 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.580285    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.580442 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.580318    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.580517 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.580494    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm2wk\" (UniqueName: \"kubernetes.io/projected/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-kube-api-access-cm2wk\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.580578 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.580543    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.580647 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.580633    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.580748 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.580661    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.580811 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.580744    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.580811 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.580778    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80939755-d4c4-4c9e-82dc-6f4349c25415-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.580925 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.580812    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvksr\" (UniqueName: \"kubernetes.io/projected/80939755-d4c4-4c9e-82dc-6f4349c25415-kube-api-access-zvksr\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.580925 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.580829    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.580925 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.580845    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.580925 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.580872    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.580925 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.580895    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.580925 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.580917    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.581230 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.580933    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.581230 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.581014    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.581230 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.581049    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.581230 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.581071    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.683203 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.683157    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.683473 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.683453    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80939755-d4c4-4c9e-82dc-6f4349c25415-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.683595 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.683579    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvksr\" (UniqueName: \"kubernetes.io/projected/80939755-d4c4-4c9e-82dc-6f4349c25415-kube-api-access-zvksr\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.683715 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.683698    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.683861 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.683845    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.683991 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.683978    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.684100 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.684088    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.684202 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.684190    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.684289 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.684277    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.684438 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.684425    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.684546 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.684533    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.684656 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.684640    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.684974 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.684956    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.685116 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.685101    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.685279 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.685263    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm2wk\" (UniqueName: \"kubernetes.io/projected/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-kube-api-access-cm2wk\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.685395 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.685381    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.685491 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.685479    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.685579 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.685567    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.686042 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.686016    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80939755-d4c4-4c9e-82dc-6f4349c25415-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.686187 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.685416    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.687038 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.687016    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.688189 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.688155    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.688608 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.688587    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.689265 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.689246    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.690736 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.690712    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.692090 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.691362    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.692090 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.691589    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.692090 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.691841    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.693877 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.693800    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.697852 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.697823    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.698317 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.698293    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.698658 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.698643    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.718457 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.718428    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.719133 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.719076    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"]
Oct 31 00:42:18.720903 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:42:18.720876    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32926f1_4427_4764_b86d_7294eaba9e0f.slice/crio-b1a414f638e1bedbe3c43909754f6a532d4920a206f9d7b356f768f4fada2f1a WatchSource:0}: Error finding container b1a414f638e1bedbe3c43909754f6a532d4920a206f9d7b356f768f4fada2f1a: Status 404 returned error can't find the container with id b1a414f638e1bedbe3c43909754f6a532d4920a206f9d7b356f768f4fada2f1a
Oct 31 00:42:18.721987 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.721969    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvksr\" (UniqueName: \"kubernetes.io/projected/80939755-d4c4-4c9e-82dc-6f4349c25415-kube-api-access-zvksr\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.722605 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.722560    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.724199 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.724166    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:42:18.724267 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.724236    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:42:18.724267 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.724263    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:42:18.730742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.730719    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm2wk\" (UniqueName: \"kubernetes.io/projected/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-kube-api-access-cm2wk\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:18.760411 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.760379    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:18.794719 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:18.794691    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:19.106845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:19.106707    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"]
Oct 31 00:42:19.108694 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:42:19.108652    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80939755_d4c4_4c9e_82dc_6f4349c25415.slice/crio-850b065f47c90352250290f1ca8f4c13db4bb2997d82ba49cd1ec2edf9018368 WatchSource:0}: Error finding container 850b065f47c90352250290f1ca8f4c13db4bb2997d82ba49cd1ec2edf9018368: Status 404 returned error can't find the container with id 850b065f47c90352250290f1ca8f4c13db4bb2997d82ba49cd1ec2edf9018368
Oct 31 00:42:19.111740 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:19.111673    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:42:19.111847 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:19.111775    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:42:19.111847 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:19.111805    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:42:19.147204 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:19.147170    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"]
Oct 31 00:42:19.156140 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:42:19.156109    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab534e5f_cfc2_4be6_acf5_bc00c9fa957f.slice/crio-e6bd84c95f2c44264f1b031384b9e70a785124ce7d3f817c308f3f9cc636cf18 WatchSource:0}: Error finding container e6bd84c95f2c44264f1b031384b9e70a785124ce7d3f817c308f3f9cc636cf18: Status 404 returned error can't find the container with id e6bd84c95f2c44264f1b031384b9e70a785124ce7d3f817c308f3f9cc636cf18
Oct 31 00:42:19.159383 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:19.159341    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:42:19.159497 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:19.159432    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:42:19.159497 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:19.159475    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:42:19.489286 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:19.489191    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch" event={"ID":"80939755-d4c4-4c9e-82dc-6f4349c25415","Type":"ContainerStarted","Data":"b0c92f0638452a6b2c72c15d0991950d63dbc7fd10ea4e7fe0fc6173ec8178eb"}
Oct 31 00:42:19.489286 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:19.489257    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch" event={"ID":"80939755-d4c4-4c9e-82dc-6f4349c25415","Type":"ContainerStarted","Data":"850b065f47c90352250290f1ca8f4c13db4bb2997d82ba49cd1ec2edf9018368"}
Oct 31 00:42:19.493349 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:19.493320    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55" event={"ID":"93505d53-b42a-4c07-ae52-d62ac15f1d79","Type":"ContainerStarted","Data":"6267638258fe3d5bb055aaeb577dbce11aeeec567f548eb786873ce50e9f9167"}
Oct 31 00:42:19.493499 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:19.493369    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55" event={"ID":"93505d53-b42a-4c07-ae52-d62ac15f1d79","Type":"ContainerStarted","Data":"e1cb77d4331d73b3ed66491f2179c82a298e599ecfbcd7e02dac02099d719f27"}
Oct 31 00:42:19.496009 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:19.495980    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp" event={"ID":"f32926f1-4427-4764-b86d-7294eaba9e0f","Type":"ContainerStarted","Data":"b5a4cc410545249d87358041f2bd130a1805239ce0da144e3bf5def222695c42"}
Oct 31 00:42:19.496153 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:19.496023    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp" event={"ID":"f32926f1-4427-4764-b86d-7294eaba9e0f","Type":"ContainerStarted","Data":"b1a414f638e1bedbe3c43909754f6a532d4920a206f9d7b356f768f4fada2f1a"}
Oct 31 00:42:19.498033 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:19.498015    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx" event={"ID":"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f","Type":"ContainerStarted","Data":"48274dd203ecd370073dda9e8f7e96b8cf8d68fc71f6bc83e57a8e3cef537205"}
Oct 31 00:42:19.498107 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:19.498050    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx" event={"ID":"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f","Type":"ContainerStarted","Data":"e6bd84c95f2c44264f1b031384b9e70a785124ce7d3f817c308f3f9cc636cf18"}
Oct 31 00:42:19.616572 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:19.616505    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx" podStartSLOduration=1.61648952 podStartE2EDuration="1.61648952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:19.614471337 +0000 UTC m=+1399.836506906" watchObservedRunningTime="2026-10-31 00:42:19.61648952 +0000 UTC m=+1399.838525059"
Oct 31 00:42:19.624924 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:19.624855    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch" podStartSLOduration=1.624834364 podStartE2EDuration="1.624834364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:19.552220556 +0000 UTC m=+1399.774256099" watchObservedRunningTime="2026-10-31 00:42:19.624834364 +0000 UTC m=+1399.846869927"
Oct 31 00:42:19.756512 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:19.756369    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp" podStartSLOduration=2.756343761 podStartE2EDuration="2.756343761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:19.737475025 +0000 UTC m=+1399.959510569" watchObservedRunningTime="2026-10-31 00:42:19.756343761 +0000 UTC m=+1399.978379306"
Oct 31 00:42:19.758485 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:19.758419    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55" podStartSLOduration=2.758400837 podStartE2EDuration="2.758400837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:19.684034544 +0000 UTC m=+1399.906070086" watchObservedRunningTime="2026-10-31 00:42:19.758400837 +0000 UTC m=+1399.980436383"
Oct 31 00:42:19.778704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:19.766752    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:19.795513 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:19.794999    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:20.228901 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:20.228859    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:20.229336 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:20.229301    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 00:42:20.240839 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:20.240783    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55" podUID="93505d53-b42a-4c07-ae52-d62ac15f1d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 00:42:20.410833 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:20.410796    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:20.413456 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:20.413423    4327 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 00:42:20.413751 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:20.413714    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp" podUID="f32926f1-4427-4764-b86d-7294eaba9e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 00:42:20.777706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:20.775673    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 00:42:20.777706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:20.775775    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch" podUID="80939755-d4c4-4c9e-82dc-6f4349c25415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 00:42:20.859703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:20.847876    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 00:42:20.859703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:20.847955    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx" podUID="ab534e5f-cfc2-4be6-acf5-bc00c9fa957f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 00:42:20.979974 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:20.977279    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:42:21.215959 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:21.214527    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 00:42:21.219693 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:21.219636    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55" podUID="93505d53-b42a-4c07-ae52-d62ac15f1d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 00:42:21.380599 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:21.380126    4327 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 00:42:21.380599 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:21.380199    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp" podUID="f32926f1-4427-4764-b86d-7294eaba9e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 00:42:21.564456 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:21.564348    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:42:21.614699 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:21.611213    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:42:21.789231 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:21.789191    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 00:42:21.811945 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:21.811895    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch" podUID="80939755-d4c4-4c9e-82dc-6f4349c25415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 00:42:21.812540 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:21.801671    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 00:42:21.812743 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:21.812720    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx" podUID="ab534e5f-cfc2-4be6-acf5-bc00c9fa957f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 00:42:21.946706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:21.946223    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:42:21.994793 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:21.994744    4327 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:42:22.223767 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:22.214304    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 00:42:22.223767 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:22.214369    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55" podUID="93505d53-b42a-4c07-ae52-d62ac15f1d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 00:42:22.378703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:22.378088    4327 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 00:42:22.378703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:22.378155    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp" podUID="f32926f1-4427-4764-b86d-7294eaba9e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 00:42:22.638082 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:22.633775    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:42:22.638082 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:22.634174    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:42:22.638082 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:22.634793    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:42:22.638082 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:22.634846    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:42:22.962869 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:22.762425    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 00:42:22.962869 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:22.762484    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch" podUID="80939755-d4c4-4c9e-82dc-6f4349c25415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 00:42:22.962869 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:22.919967    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 00:42:22.962869 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:22.920043    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx" podUID="ab534e5f-cfc2-4be6-acf5-bc00c9fa957f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 00:42:22.962869 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:22.960783    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:42:22.962869 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:22.961282    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:42:22.985252 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:22.985206    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:42:23.231997 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:23.206801    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:42:23.266968 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:23.232746    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 00:42:23.290111 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:23.288777    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:42:23.315706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:23.294785    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55" podUID="93505d53-b42a-4c07-ae52-d62ac15f1d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 00:42:23.411851 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:23.411776    4327 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 00:42:23.412823 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:23.412787    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp" podUID="f32926f1-4427-4764-b86d-7294eaba9e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 00:42:23.644704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:23.637009    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:42:23.785903 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:23.785866    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 00:42:23.786183 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:23.786151    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch" podUID="80939755-d4c4-4c9e-82dc-6f4349c25415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 00:42:23.848595 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:23.848563    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 00:42:23.848918 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:23.848871    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx" podUID="ab534e5f-cfc2-4be6-acf5-bc00c9fa957f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 00:42:24.340032 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:24.339997    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 00:42:24.390705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:24.366190    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55" podUID="93505d53-b42a-4c07-ae52-d62ac15f1d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 00:42:24.403898 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:24.366793    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 00:42:24.406295 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:24.406048    4327 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 00:42:24.406295 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:24.406101    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp" podUID="f32926f1-4427-4764-b86d-7294eaba9e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 00:42:24.672643 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:24.672562    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 00:42:24.761144 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:24.761105    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 00:42:24.761532 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:24.761498    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch" podUID="80939755-d4c4-4c9e-82dc-6f4349c25415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 00:42:24.807981 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:24.807940    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 00:42:24.808270 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:24.808241    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx" podUID="ab534e5f-cfc2-4be6-acf5-bc00c9fa957f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 00:42:25.073331 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:25.072648    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:42:25.214095 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:25.213568    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 00:42:25.214095 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:25.213626    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55" podUID="93505d53-b42a-4c07-ae52-d62ac15f1d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 00:42:25.385704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:25.384783    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:42:25.386514 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:25.386485    4327 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 00:42:25.387134 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:25.387105    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp" podUID="f32926f1-4427-4764-b86d-7294eaba9e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 00:42:25.621592 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:25.621250    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:42:25.674901 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:25.673776    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:42:25.761699 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:25.760766    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 00:42:25.761699 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:25.760830    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch" podUID="80939755-d4c4-4c9e-82dc-6f4349c25415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 00:42:25.799868 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:25.796853    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 00:42:25.799868 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:25.796917    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx" podUID="ab534e5f-cfc2-4be6-acf5-bc00c9fa957f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 00:42:26.096807 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:26.096704    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:42:26.211708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:26.180975    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:42:26.244706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:26.213556    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 00:42:26.244706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:26.213611    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55" podUID="93505d53-b42a-4c07-ae52-d62ac15f1d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 00:42:26.391612 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:26.391562    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 00:42:26.392809 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:26.392786    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:42:26.561709 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:26.561268    4327 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 00:42:26.561709 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:26.561330    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp" podUID="f32926f1-4427-4764-b86d-7294eaba9e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 00:42:26.728701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:26.727901    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:42:26.776527 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:26.776486    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 00:42:26.776833 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:26.776801    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch" podUID="80939755-d4c4-4c9e-82dc-6f4349c25415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 00:42:26.792714 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:26.792662    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:42:26.805640 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:26.805610    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:42:26.805904 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:26.795789    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 00:42:26.806054 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:26.806026    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx" podUID="ab534e5f-cfc2-4be6-acf5-bc00c9fa957f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 00:42:26.810079 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:26.810049    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:42:27.215387 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:27.215351    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 00:42:27.215588 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:27.215419    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55" podUID="93505d53-b42a-4c07-ae52-d62ac15f1d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 00:42:27.379803 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:27.379084    4327 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 00:42:27.379803 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:27.379161    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp" podUID="f32926f1-4427-4764-b86d-7294eaba9e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 00:42:27.770704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:27.767011    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 00:42:27.770704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:27.767077    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch" podUID="80939755-d4c4-4c9e-82dc-6f4349c25415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 00:42:27.806708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:27.806374    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 00:42:27.806708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:27.806433    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx" podUID="ab534e5f-cfc2-4be6-acf5-bc00c9fa957f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 00:42:28.222608 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:28.221786    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:28.223198 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:28.223121    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:28.224711 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:28.224610    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:42:28.429028 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:28.428995    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:28.430671 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:28.430643    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:28.456802 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:28.456023    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 00:42:28.800371 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:28.800338    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:28.801297 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:28.801278    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:28.801977 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:28.801950    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:42:29.062858 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:29.062773    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:29.064744 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:29.064721    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:29.066160 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:29.066137    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:42:30.986779 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:30.986743    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:42:30.998446 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:30.998422    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:42:32.469062 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.469026    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"]
Oct 31 00:42:32.474794 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.474768    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.517114 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.517085    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"]
Oct 31 00:42:32.560665 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.560624    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.560665 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.560701    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.560906 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.560810    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.560906 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.560843    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq2z2\" (UniqueName: \"kubernetes.io/projected/ed99fa70-689b-472e-8e25-62e70c3f2972-kube-api-access-pq2z2\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.560976 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.560912    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed99fa70-689b-472e-8e25-62e70c3f2972-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.561012 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.561002    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.561066 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.561051    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.561102 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.561073    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.561235 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.561216    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.662398 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.662358    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.662576 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.662436    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.662576 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.662487    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.662576 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.662516    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq2z2\" (UniqueName: \"kubernetes.io/projected/ed99fa70-689b-472e-8e25-62e70c3f2972-kube-api-access-pq2z2\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.662733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.662582    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed99fa70-689b-472e-8e25-62e70c3f2972-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.662733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.662714    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.662853 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.662807    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.662853 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.662839    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.662968 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.662949    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.663031 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.662953    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.663168 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.663144    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.663257 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.663233    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.663318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.663285    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.663459 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.663436    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed99fa70-689b-472e-8e25-62e70c3f2972-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.665046 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.665024    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.665125 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.665029    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.686692 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.686658    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.686828 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.686756    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq2z2\" (UniqueName: \"kubernetes.io/projected/ed99fa70-689b-472e-8e25-62e70c3f2972-kube-api-access-pq2z2\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:32.825149 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:32.825063    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:33.148109 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:33.148080    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"]
Oct 31 00:42:33.149618 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:42:33.149593    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded99fa70_689b_472e_8e25_62e70c3f2972.slice/crio-b405b0ac22fc86dcf476df2907768675da3e5f82e710db143551a3a9c6e12efb WatchSource:0}: Error finding container b405b0ac22fc86dcf476df2907768675da3e5f82e710db143551a3a9c6e12efb: Status 404 returned error can't find the container with id b405b0ac22fc86dcf476df2907768675da3e5f82e710db143551a3a9c6e12efb
Oct 31 00:42:33.152153 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:33.152122    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:42:33.152227 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:33.152188    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:42:33.152227 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:33.152214    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:42:33.633994 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:33.633750    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58" event={"ID":"ed99fa70-689b-472e-8e25-62e70c3f2972","Type":"ContainerStarted","Data":"46e9236e088d6783f24a0109d50cf8c0c2c4a666f91d35db389ac0e7db70f1ff"}
Oct 31 00:42:33.633994 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:33.633812    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58" event={"ID":"ed99fa70-689b-472e-8e25-62e70c3f2972","Type":"ContainerStarted","Data":"b405b0ac22fc86dcf476df2907768675da3e5f82e710db143551a3a9c6e12efb"}
Oct 31 00:42:33.705700 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:33.703241    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58" podStartSLOduration=1.7032217379999999 podStartE2EDuration="1.703221738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:33.703090012 +0000 UTC m=+1413.925125555" watchObservedRunningTime="2026-10-31 00:42:33.703221738 +0000 UTC m=+1413.925257282"
Oct 31 00:42:33.831854 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:33.831816    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:34.851872 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:34.851831    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 00:42:34.853460 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:34.853422    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58" podUID="ed99fa70-689b-472e-8e25-62e70c3f2972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 00:42:35.835128 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:35.835084    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 00:42:35.835415 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:35.835386    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58" podUID="ed99fa70-689b-472e-8e25-62e70c3f2972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 00:42:36.829704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:36.826124    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 00:42:36.829704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:36.826203    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58" podUID="ed99fa70-689b-472e-8e25-62e70c3f2972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 00:42:37.834703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:37.832974    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 00:42:37.834703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:37.833057    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58" podUID="ed99fa70-689b-472e-8e25-62e70c3f2972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 00:42:38.838703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:38.832084    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 00:42:38.838703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:38.832160    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58" podUID="ed99fa70-689b-472e-8e25-62e70c3f2972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 00:42:39.825581 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:39.825544    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 00:42:39.825789 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:39.825634    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58" podUID="ed99fa70-689b-472e-8e25-62e70c3f2972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 00:42:40.830371 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:40.830341    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:40.830768 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:40.830711    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:40.831548 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:40.831527    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:42:45.300515 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:42:45.300475    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:42:55.076743 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.076710    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"]
Oct 31 00:42:55.081798 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.081773    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:42:55.105976 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.105950    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c9e377\"/\"distinct-2-w5-c9e377-kserve-self-signed-certs\""
Oct 31 00:42:55.121844 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.121814    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"]
Oct 31 00:42:55.191527 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.191488    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/378b2c59-105c-4a30-82cc-4e0d1716414d-tls-certs\") pod \"distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv\" (UID: \"378b2c59-105c-4a30-82cc-4e0d1716414d\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:42:55.191527 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.191536    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-dshm\") pod \"distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv\" (UID: \"378b2c59-105c-4a30-82cc-4e0d1716414d\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:42:55.191772 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.191614    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7cm4\" (UniqueName: \"kubernetes.io/projected/378b2c59-105c-4a30-82cc-4e0d1716414d-kube-api-access-x7cm4\") pod \"distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv\" (UID: \"378b2c59-105c-4a30-82cc-4e0d1716414d\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:42:55.191856 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.191835    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-model-cache\") pod \"distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv\" (UID: \"378b2c59-105c-4a30-82cc-4e0d1716414d\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:42:55.191899 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.191885    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-home\") pod \"distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv\" (UID: \"378b2c59-105c-4a30-82cc-4e0d1716414d\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:42:55.191963 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.191946    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-tmp-dir\") pod \"distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv\" (UID: \"378b2c59-105c-4a30-82cc-4e0d1716414d\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:42:55.307779 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.307738    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-model-cache\") pod \"distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv\" (UID: \"378b2c59-105c-4a30-82cc-4e0d1716414d\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:42:55.308113 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.308083    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-home\") pod \"distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv\" (UID: \"378b2c59-105c-4a30-82cc-4e0d1716414d\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:42:55.308277 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.308256    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-tmp-dir\") pod \"distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv\" (UID: \"378b2c59-105c-4a30-82cc-4e0d1716414d\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:42:55.308613 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.308589    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/378b2c59-105c-4a30-82cc-4e0d1716414d-tls-certs\") pod \"distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv\" (UID: \"378b2c59-105c-4a30-82cc-4e0d1716414d\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:42:55.334217 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.309130    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-home\") pod \"distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv\" (UID: \"378b2c59-105c-4a30-82cc-4e0d1716414d\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:42:55.334510 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.334082    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-dshm\") pod \"distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv\" (UID: \"378b2c59-105c-4a30-82cc-4e0d1716414d\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:42:55.334733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.334711    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7cm4\" (UniqueName: \"kubernetes.io/projected/378b2c59-105c-4a30-82cc-4e0d1716414d-kube-api-access-x7cm4\") pod \"distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv\" (UID: \"378b2c59-105c-4a30-82cc-4e0d1716414d\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:42:55.336013 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.335987    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-tmp-dir\") pod \"distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv\" (UID: \"378b2c59-105c-4a30-82cc-4e0d1716414d\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:42:55.336341 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.336322    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-model-cache\") pod \"distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv\" (UID: \"378b2c59-105c-4a30-82cc-4e0d1716414d\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:42:55.361290 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.360796    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-dshm\") pod \"distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv\" (UID: \"378b2c59-105c-4a30-82cc-4e0d1716414d\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:42:55.372716 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.366136    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7cm4\" (UniqueName: \"kubernetes.io/projected/378b2c59-105c-4a30-82cc-4e0d1716414d-kube-api-access-x7cm4\") pod \"distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv\" (UID: \"378b2c59-105c-4a30-82cc-4e0d1716414d\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:42:55.379433 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.379399    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/378b2c59-105c-4a30-82cc-4e0d1716414d-tls-certs\") pod \"distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv\" (UID: \"378b2c59-105c-4a30-82cc-4e0d1716414d\") " pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:42:55.398890 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.398572    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:42:55.780586 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.780556    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"]
Oct 31 00:42:55.782213 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:42:55.782185    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod378b2c59_105c_4a30_82cc_4e0d1716414d.slice/crio-5427d5fc0404d61b52be1c90eedcb2faa9c48e2812045bf03da2133b4536b645 WatchSource:0}: Error finding container 5427d5fc0404d61b52be1c90eedcb2faa9c48e2812045bf03da2133b4536b645: Status 404 returned error can't find the container with id 5427d5fc0404d61b52be1c90eedcb2faa9c48e2812045bf03da2133b4536b645
Oct 31 00:42:55.862565 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.862533    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" event={"ID":"378b2c59-105c-4a30-82cc-4e0d1716414d","Type":"ContainerStarted","Data":"5a47e0a6db1aaf5592f4f71c7062a302d32ec8ba8883eac15e7b16e6b146de9d"}
Oct 31 00:42:55.862737 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.862578    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" event={"ID":"378b2c59-105c-4a30-82cc-4e0d1716414d","Type":"ContainerStarted","Data":"5427d5fc0404d61b52be1c90eedcb2faa9c48e2812045bf03da2133b4536b645"}
Oct 31 00:42:55.907784 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:55.907728    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podStartSLOduration=0.907710049 podStartE2EDuration="907.710049ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:55.904447474 +0000 UTC m=+1436.126483020" watchObservedRunningTime="2026-10-31 00:42:55.907710049 +0000 UTC m=+1436.129745593"
Oct 31 00:42:56.399938 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:42:56.399907    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:43:02.720567 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:02.720505    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:02.827706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:02.826241    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 00:43:02.872706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:02.849930    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:43:03.006868 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.006759    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:03.227702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.215804    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:03.250022 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.245222    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"]
Oct 31 00:43:03.266209 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.266124    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.390781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.390739    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"]
Oct 31 00:43:03.394663 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.394625    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.400705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.394941    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.400945 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.400917    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.401084 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.401067    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.401282 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.401266    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.406708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.401527    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb95m\" (UniqueName: \"kubernetes.io/projected/2bd889fe-02e5-45e4-aaa6-156d3beabf00-kube-api-access-qb95m\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.406928 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.406904    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.407184 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.407163    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.414714 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.407354    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.639865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.530505    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.639865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.530580    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.639865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.530653    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.639865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.530695    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.639865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.530721    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.639865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.530821    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.639865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.626775    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.662278 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.662245    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"]
Oct 31 00:43:03.662611 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.662592    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:03.720703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.694844    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb95m\" (UniqueName: \"kubernetes.io/projected/2bd889fe-02e5-45e4-aaa6-156d3beabf00-kube-api-access-qb95m\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.720703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.694909    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.720703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.695000    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.734925 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.722145    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.734925 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.722830    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.734925 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.723072    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.734925 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.724964    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 00:43:03.735769 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.735743    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:43:03.761749 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.745246    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.793929 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.791962    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.834348 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.834303    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:03.836397 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.836371    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:43:03.865767 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.856277    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:03.865767 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.856559    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:43:03.865767 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.858923    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:03.865767 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.859001    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:03.865767 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.859079    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1acb532-b591-421f-87a7-61fbaec51ed9-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:03.865767 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.859147    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:03.865767 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.859309    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w69kz\" (UniqueName: \"kubernetes.io/projected/b1acb532-b591-421f-87a7-61fbaec51ed9-kube-api-access-w69kz\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:03.865767 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.859386    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:03.865767 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.859412    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:03.865767 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.859468    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:03.865767 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.859549    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:03.898473 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.892361    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"]
Oct 31 00:43:03.909476 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.909440    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.932867 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.920324    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:03.998721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.994893    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:03.998721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.994998    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:03.998721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.995086    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:03.998721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.995144    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:03.998721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.995231    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1acb532-b591-421f-87a7-61fbaec51ed9-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:03.998721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.995295    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:03.998721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.995480    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w69kz\" (UniqueName: \"kubernetes.io/projected/b1acb532-b591-421f-87a7-61fbaec51ed9-kube-api-access-w69kz\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:03.998721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.995569    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:03.998721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:03.995593    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:04.054705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:04.037971    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:04.054705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:04.038304    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:04.093897 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:04.082947    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:04.093897 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:04.083178    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:04.109753 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:04.104943    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:04.138098 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:04.138058    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1acb532-b591-421f-87a7-61fbaec51ed9-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:04.176822 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:04.152874    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb95m\" (UniqueName: \"kubernetes.io/projected/2bd889fe-02e5-45e4-aaa6-156d3beabf00-kube-api-access-qb95m\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:04.316085 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:04.316041    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:04.386742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:04.357364    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:04.424720 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:04.405907    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:04.446408 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:04.446304    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"]
Oct 31 00:43:04.482272 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:04.482199    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w69kz\" (UniqueName: \"kubernetes.io/projected/b1acb532-b591-421f-87a7-61fbaec51ed9-kube-api-access-w69kz\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:04.545875 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:04.545843    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:04.647917 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:04.645333    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"]
Oct 31 00:43:04.647917 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:04.645437    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:43:04.647917 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:04.645483    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:43:04.647917 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:04.645518    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:43:04.647917 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:04.646041    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:04.757262 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:04.726755    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:04.757262 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:04.734355    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:04.882053 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:04.882003    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:43:05.022812 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.008592    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:05.022812 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.008715    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:05.022812 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.008829    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:05.022812 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.008866    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:05.022812 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.008949    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:05.022812 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.009050    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:05.022812 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.009079    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvn2f\" (UniqueName: \"kubernetes.io/projected/dc70d4a7-43f4-4a03-883b-2035e3f2c137-kube-api-access-zvn2f\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:05.022812 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.009113    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:05.022812 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.009300    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:05.165752 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.038982    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:05.165752 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.051219    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 00:43:05.165752 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.138816    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:05.165752 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.138872    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvn2f\" (UniqueName: \"kubernetes.io/projected/dc70d4a7-43f4-4a03-883b-2035e3f2c137-kube-api-access-zvn2f\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:05.165752 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.138908    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:05.165752 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.139057    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:05.165752 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.139148    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:05.165752 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.139203    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:05.165752 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.139292    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:05.165752 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.139330    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:05.165752 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.139395    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:05.173424 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.173044    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:05.297902 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.297808    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:05.299695 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.299637    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:05.300108 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.300088    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:05.300466 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.300445    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:05.300876 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.300856    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:05.357542 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.357501    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:05.593752 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.583062    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:05.593752 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.584547    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:05.637931 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.617093    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvn2f\" (UniqueName: \"kubernetes.io/projected/dc70d4a7-43f4-4a03-883b-2035e3f2c137-kube-api-access-zvn2f\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:06.021706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:05.965423    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 00:43:06.062364 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:06.062314    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:06.076857 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:06.074708    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:06.171763 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:06.171648    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj" podUID="b1988c46-0b7f-4edd-b441-ab408ad0a246" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 00:43:06.251249 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:06.251207    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 00:43:06.253584 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:06.253556    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:43:06.367720 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:06.358436    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:06.460333 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:06.457574    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:43:06.460333 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:06.457914    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded"
Oct 31 00:43:06.575285 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:06.575251    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:43:06.575490 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:06.575318    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:43:06.581413 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:06.581387    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:43:06.581932 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:06.581903    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:43:06.587267 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:06.587241    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:43:06.729096 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:06.729069    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"]
Oct 31 00:43:06.732114 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:43:06.732082    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd889fe_02e5_45e4_aaa6_156d3beabf00.slice/crio-bff79a93dfed6c13fb8cdd6ef86923fe40af48887d0af30967e4c4b9d77559f2 WatchSource:0}: Error finding container bff79a93dfed6c13fb8cdd6ef86923fe40af48887d0af30967e4c4b9d77559f2: Status 404 returned error can't find the container with id bff79a93dfed6c13fb8cdd6ef86923fe40af48887d0af30967e4c4b9d77559f2
Oct 31 00:43:06.747715 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:06.747641    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:43:06.747834 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:06.747769    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:43:06.747881 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:06.747827    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:43:06.980804 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:06.980761    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"]
Oct 31 00:43:06.984384 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:06.984348    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:43:06.984510 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:06.984432    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:43:06.984510 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:06.984481    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:43:06.984626 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:06.984536    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:43:06.984626 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:06.984616    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:43:06.984744 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:06.984693    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:43:06.984744 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:06.984699    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"]
Oct 31 00:43:07.413016 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:07.412979    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:43:07.413477 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:07.413024    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr" event={"ID":"dc70d4a7-43f4-4a03-883b-2035e3f2c137","Type":"ContainerStarted","Data":"afdc6044c6f2475b0a04c55a3da19a2cef18cc858d5c6902f9ec24bfbd3d6b6b"}
Oct 31 00:43:07.413477 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:07.413063    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr" event={"ID":"dc70d4a7-43f4-4a03-883b-2035e3f2c137","Type":"ContainerStarted","Data":"6e5a74c64e4f5070f3067b5f979a8188c309774b42b54e1bbc8b4ee0f39c1807"}
Oct 31 00:43:07.415100 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:07.415071    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh" event={"ID":"b1acb532-b591-421f-87a7-61fbaec51ed9","Type":"ContainerStarted","Data":"481c2e052188b2d9ed775b0ff9cca511afbaeedb2ce615a9174dc6da7a1f22ee"}
Oct 31 00:43:07.415225 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:07.415125    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh" event={"ID":"b1acb532-b591-421f-87a7-61fbaec51ed9","Type":"ContainerStarted","Data":"0f332d19f9e2ef8b4a1aae74e65bd203249697636980ea765fa7edf18d432829"}
Oct 31 00:43:07.416963 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:07.416939    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg" event={"ID":"2bd889fe-02e5-45e4-aaa6-156d3beabf00","Type":"ContainerStarted","Data":"239a4a654071d146230cda27adc28746a87e54465c0bebda4a813aafc7287230"}
Oct 31 00:43:07.417080 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:07.416982    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg" event={"ID":"2bd889fe-02e5-45e4-aaa6-156d3beabf00","Type":"ContainerStarted","Data":"bff79a93dfed6c13fb8cdd6ef86923fe40af48887d0af30967e4c4b9d77559f2"}
Oct 31 00:43:07.507547 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:07.507496    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh" podStartSLOduration=4.50748196 podStartE2EDuration="4.50748196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:07.503942044 +0000 UTC m=+1447.725977586" watchObservedRunningTime="2026-10-31 00:43:07.50748196 +0000 UTC m=+1447.729517573"
Oct 31 00:43:07.554420 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:07.554361    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg" podStartSLOduration=4.554346532 podStartE2EDuration="4.554346532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:07.550887793 +0000 UTC m=+1447.772923336" watchObservedRunningTime="2026-10-31 00:43:07.554346532 +0000 UTC m=+1447.776382077"
Oct 31 00:43:07.601833 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:07.601768    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr" podStartSLOduration=4.601744168 podStartE2EDuration="4.601744168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:07.59753554 +0000 UTC m=+1447.819571083" watchObservedRunningTime="2026-10-31 00:43:07.601744168 +0000 UTC m=+1447.823779711"
Oct 31 00:43:08.085704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:08.085444    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:08.098709 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:08.096535    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 00:43:08.098709 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:08.096612    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh" podUID="b1acb532-b591-421f-87a7-61fbaec51ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 00:43:08.307043 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:08.307009    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:08.308224 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:08.308190    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 00:43:08.308279 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:08.308249    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr" podUID="dc70d4a7-43f4-4a03-883b-2035e3f2c137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 00:43:08.337765 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:08.330980    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:08.337765 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:08.331904    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 00:43:08.337765 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:08.331953    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg" podUID="2bd889fe-02e5-45e4-aaa6-156d3beabf00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 00:43:09.100762 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:09.096818    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 00:43:09.100762 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:09.096884    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh" podUID="b1acb532-b591-421f-87a7-61fbaec51ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 00:43:09.306403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:09.306365    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 00:43:09.306573 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:09.306418    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr" podUID="dc70d4a7-43f4-4a03-883b-2035e3f2c137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 00:43:09.321058 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:09.321026    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 00:43:09.321292 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:09.321267    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg" podUID="2bd889fe-02e5-45e4-aaa6-156d3beabf00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 00:43:10.082702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:10.082087    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 00:43:10.082702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:10.082151    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh" podUID="b1acb532-b591-421f-87a7-61fbaec51ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 00:43:10.307699 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:10.307334    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 00:43:10.307699 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:10.307388    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr" podUID="dc70d4a7-43f4-4a03-883b-2035e3f2c137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 00:43:10.398902 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:10.398873    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 00:43:10.399964 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:10.399778    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg" podUID="2bd889fe-02e5-45e4-aaa6-156d3beabf00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 00:43:11.092705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:11.092058    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 00:43:11.092705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:11.092124    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh" podUID="b1acb532-b591-421f-87a7-61fbaec51ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 00:43:11.307086 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:11.307052    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 00:43:11.307260 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:11.307106    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr" podUID="dc70d4a7-43f4-4a03-883b-2035e3f2c137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 00:43:11.345782 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:11.345092    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 00:43:11.345782 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:11.345148    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg" podUID="2bd889fe-02e5-45e4-aaa6-156d3beabf00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 00:43:12.084496 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:12.083957    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 00:43:12.103715 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:12.103649    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh" podUID="b1acb532-b591-421f-87a7-61fbaec51ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 00:43:12.306854 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:12.306775    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 00:43:12.307047 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:12.306849    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr" podUID="dc70d4a7-43f4-4a03-883b-2035e3f2c137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 00:43:12.415547 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:12.405423    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 00:43:12.415547 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:12.405489    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg" podUID="2bd889fe-02e5-45e4-aaa6-156d3beabf00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 00:43:13.097938 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:13.097447    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 00:43:13.097938 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:13.097514    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh" podUID="b1acb532-b591-421f-87a7-61fbaec51ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 00:43:13.309031 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:13.308989    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 00:43:13.309202 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:13.309054    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr" podUID="dc70d4a7-43f4-4a03-883b-2035e3f2c137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 00:43:13.343705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:13.339730    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 00:43:13.343705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:13.339792    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg" podUID="2bd889fe-02e5-45e4-aaa6-156d3beabf00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 00:43:14.079954 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:14.079906    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 00:43:14.080453 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:14.080024    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh" podUID="b1acb532-b591-421f-87a7-61fbaec51ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 00:43:14.306518 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:14.306468    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 00:43:14.306766 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:14.306550    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr" podUID="dc70d4a7-43f4-4a03-883b-2035e3f2c137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 00:43:14.317065 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:14.317019    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 00:43:14.317221 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:14.317085    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg" podUID="2bd889fe-02e5-45e4-aaa6-156d3beabf00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 00:43:15.347274 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:15.347228    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:15.349046 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:15.349023    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:15.350665 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:15.350619    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:43:15.382786 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:43:15.382735    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:43:15.539881 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:15.539845    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:15.541049 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:15.541023    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:15.542825 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:15.542803    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:43:15.550296 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:15.550274    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:15.550764 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:15.550714    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:15.552547 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:15.552525    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:43:17.969850 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:17.969815    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"]
Oct 31 00:43:17.976136 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:17.976106    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.000162 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.000133    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"]
Oct 31 00:43:18.093387 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.093341    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.093590 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.093473    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.093590 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.093520    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.093760 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.093634    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.093760 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.093713    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcfwb\" (UniqueName: \"kubernetes.io/projected/ee820ea1-a991-40f1-a2f5-b3a9519bc885-kube-api-access-tcfwb\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.093760 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.093743    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.093930 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.093773    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.093930 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.093868    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.094050 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.093933    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.195616 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.195582    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.195831 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.195637    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.195831 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.195738    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.195831 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.195778    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcfwb\" (UniqueName: \"kubernetes.io/projected/ee820ea1-a991-40f1-a2f5-b3a9519bc885-kube-api-access-tcfwb\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.195831 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.195800    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.196049 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.195837    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.196049 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.195990    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.196160 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.196102    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.196160 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.196132    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.196266 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.196188    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.196435 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.196413    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.196518 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.196473    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.196518 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.196473    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.196595 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.196512    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.198004 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.197984    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.198285 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.198267    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.222500 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.222431    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcfwb\" (UniqueName: \"kubernetes.io/projected/ee820ea1-a991-40f1-a2f5-b3a9519bc885-kube-api-access-tcfwb\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.222729 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.222710    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.294537 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.294503    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:18.617893 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.617866    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"]
Oct 31 00:43:18.618815 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:43:18.618788    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee820ea1_a991_40f1_a2f5_b3a9519bc885.slice/crio-1e72807d2434ad479dbbb6fe76e41cccbc2b3b15ddd1f229e4ef2ad05a8ae72f WatchSource:0}: Error finding container 1e72807d2434ad479dbbb6fe76e41cccbc2b3b15ddd1f229e4ef2ad05a8ae72f: Status 404 returned error can't find the container with id 1e72807d2434ad479dbbb6fe76e41cccbc2b3b15ddd1f229e4ef2ad05a8ae72f
Oct 31 00:43:18.621599 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.621562    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:43:18.621727 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.621645    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:43:18.621727 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:18.621710    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:43:19.094193 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:19.094098    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt" event={"ID":"ee820ea1-a991-40f1-a2f5-b3a9519bc885","Type":"ContainerStarted","Data":"cc55c21d97152e65dee00c9b2b73adb222c0907d357088d4d7a1751abe64ada3"}
Oct 31 00:43:19.094193 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:19.094185    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt" event={"ID":"ee820ea1-a991-40f1-a2f5-b3a9519bc885","Type":"ContainerStarted","Data":"1e72807d2434ad479dbbb6fe76e41cccbc2b3b15ddd1f229e4ef2ad05a8ae72f"}
Oct 31 00:43:19.140151 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:19.140096    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt" podStartSLOduration=2.14008105 podStartE2EDuration="2.14008105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:19.138918682 +0000 UTC m=+1459.360954224" watchObservedRunningTime="2026-10-31 00:43:19.14008105 +0000 UTC m=+1459.362116593"
Oct 31 00:43:19.294652 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:19.294614    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:19.296449 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:19.296417    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 00:43:19.296603 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:19.296486    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt" podUID="ee820ea1-a991-40f1-a2f5-b3a9519bc885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 00:43:20.295888 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:20.295849    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 00:43:20.296345 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:20.295927    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt" podUID="ee820ea1-a991-40f1-a2f5-b3a9519bc885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 00:43:21.295305 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:21.295270    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 00:43:21.295477 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:21.295321    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt" podUID="ee820ea1-a991-40f1-a2f5-b3a9519bc885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 00:43:22.295390 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:22.295352    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 00:43:22.297928 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:22.295414    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt" podUID="ee820ea1-a991-40f1-a2f5-b3a9519bc885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 00:43:23.295431 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:23.295396    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 00:43:23.295926 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:23.295449    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt" podUID="ee820ea1-a991-40f1-a2f5-b3a9519bc885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 00:43:24.301877 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:24.301840    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 00:43:24.302485 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:24.302437    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt" podUID="ee820ea1-a991-40f1-a2f5-b3a9519bc885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 00:43:25.295007 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:25.294947    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 00:43:25.295172 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:25.295027    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt" podUID="ee820ea1-a991-40f1-a2f5-b3a9519bc885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 00:43:26.299251 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:26.299221    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:26.300368 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:26.299601    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:26.300368 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:26.300333    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:43:34.363906 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:34.363858    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:34.637962 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:34.637840    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:34.734838 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:34.734782    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:34.921756 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:34.917075    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:35.064728 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:35.064613    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:35.623380 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:35.623330    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:35.641751 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:35.640784    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:35.641751 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:35.641315    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:43:35.737831 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:35.737785    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:35.765706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:35.742648    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:43:36.069640 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:36.068802    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:36.285799 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:36.284733    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:43:36.430705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:36.407206    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:36.430705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:36.407563    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:43:36.430705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:36.408953    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 00:43:36.643707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:36.642791    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:36.775195 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:36.775089    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:37.012180 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:37.012140    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:43:37.086630 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:37.086523    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:37.282711 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:37.281042    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:37.483592 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:37.483548    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:43:37.539962 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:37.538287    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:37.541174 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:37.540259    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:43:37.541422 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:37.541407    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:43:37.680784 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:37.643549    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:37.944704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:37.866230    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:38.153563 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:38.153503    4327 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:43:38.168075 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:38.168015    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6" podUID="14093068-2225-44ec-9ec6-9f85d5beeb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:38.185783 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:38.185732    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:43:38.255954 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:38.255849    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:38.549017 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:38.548903    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:38.658738 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:38.594032    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 00:43:38.957984 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:38.957933    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:38.959072 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:38.959035    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 00:43:39.245296 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:39.245174    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:43:39.304197 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:39.304155    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:39.469549 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:43:39.468532    4327 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.234s"
Oct 31 00:43:39.471671 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:39.470697    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:39.486984 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:39.486950    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:43:39.669775 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:39.669731    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:39.670445 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:39.670417    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:43:40.086768 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:40.061746    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 00:43:40.086768 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:40.063414    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 00:43:40.245891 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:40.245845    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:43:40.319722 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:40.307519    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:40.417093 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:40.417032    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 00:43:40.674220 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:40.674112    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:40.675697 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:40.674493    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 00:43:41.065511 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:41.065414    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:41.066145 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:41.066114    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 00:43:41.266508 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:41.266465    4327 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:43:41.267195 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:41.267144    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh" podUID="2a45ba96-dddd-43f5-9e50-0b08194799f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:41.290043 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:41.289996    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:41.335946 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:41.335826    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:43:41.336272 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:41.336195    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5" podUID="dbb151f9-f859-4faf-a830-9da6730837c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:41.702921 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:41.702871    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:41.721551 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:41.703233    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:41.721808 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:41.703728    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:41.832815 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:41.832773    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:43:41.833081 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:41.833051    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8" podUID="c9c74e25-811d-414f-b093-a907ca85631d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:42.032988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:42.032808    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:42.070205 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:42.067414    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:42.070205 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:42.067740    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 00:43:42.302901 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:42.302787    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:42.590301 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:42.589954    4327 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:43:42.590301 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:42.590055    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd" podUID="7b9657eb-33b5-4847-9ce7-39085e57f7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:42.621814 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:42.621761    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:43:42.764703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:42.728592    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:42.789121 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:42.747794    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 00:43:42.790219 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:42.747848    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:43:43.114179 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:43.114131    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 00:43:43.125543 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:43.114742    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:43.291877 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:43.291844    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:43:43.316342 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:43.312606    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:43:43.409630 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:43.409544    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:43:43.411049 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:43.409878    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:43:43.416028 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:43.415994    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:43:45.294341 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:43:45.294305    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:43:47.695789 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.695752    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"]
Oct 31 00:43:47.704744 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.704715    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.726276 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.726237    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.726448 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.726300    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.726522 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.726441    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.726522 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.726499    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.726650 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.726624    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/148ce8dc-1145-488d-940e-111d5cb3d9fc-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.726796 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.726777    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56cqt\" (UniqueName: \"kubernetes.io/projected/148ce8dc-1145-488d-940e-111d5cb3d9fc-kube-api-access-56cqt\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.726875 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.726820    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.726936 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.726902    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.727046 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.727028    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.728820 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.728800    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"]
Oct 31 00:43:47.828668 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.828625    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/148ce8dc-1145-488d-940e-111d5cb3d9fc-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.828903 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.828758    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56cqt\" (UniqueName: \"kubernetes.io/projected/148ce8dc-1145-488d-940e-111d5cb3d9fc-kube-api-access-56cqt\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.828903 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.828788    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.828903 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.828812    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.828903 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.828899    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.829126 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.828961    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.829126 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.828991    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.829126 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.829069    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.829126 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.829111    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.829377 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.829271    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.829377 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.829306    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.829486 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.829389    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.829548 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.829511    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/148ce8dc-1145-488d-940e-111d5cb3d9fc-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.829548 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.829512    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.831189 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.831167    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.831524 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.831501    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.852911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.852880    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:47.853154 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:47.853130    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56cqt\" (UniqueName: \"kubernetes.io/projected/148ce8dc-1145-488d-940e-111d5cb3d9fc-kube-api-access-56cqt\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:48.026525 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:48.026430    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:48.401338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:48.396832    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"]
Oct 31 00:43:48.416606 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:48.416528    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:43:48.417150 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:48.417072    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:43:48.417431 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:48.417410    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:43:48.543962 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:48.543930    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj" event={"ID":"148ce8dc-1145-488d-940e-111d5cb3d9fc","Type":"ContainerStarted","Data":"34380236d989edbd1abba0ac7eaf05175fd072a6df678f9294da40787f3ff53f"}
Oct 31 00:43:49.553233 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:49.553195    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj" event={"ID":"148ce8dc-1145-488d-940e-111d5cb3d9fc","Type":"ContainerStarted","Data":"56bfc919e76e2fe0ec2eb4387ebc5f72fc19cce0225ae22f678fb3c7ad97f710"}
Oct 31 00:43:49.614332 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:49.614274    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj" podStartSLOduration=2.61425668 podStartE2EDuration="2.61425668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:49.611170109 +0000 UTC m=+1489.833205662" watchObservedRunningTime="2026-10-31 00:43:49.61425668 +0000 UTC m=+1489.836292222"
Oct 31 00:43:50.028874 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:50.028789    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:50.029909 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:50.029553    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 00:43:50.029909 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:50.029608    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj" podUID="148ce8dc-1145-488d-940e-111d5cb3d9fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 00:43:51.036705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:51.030795    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 00:43:51.036705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:51.030859    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj" podUID="148ce8dc-1145-488d-940e-111d5cb3d9fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 00:43:52.031701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:52.030064    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 00:43:52.031701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:52.030128    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj" podUID="148ce8dc-1145-488d-940e-111d5cb3d9fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 00:43:53.043850 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:53.043806    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 00:43:53.044374 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:53.044340    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj" podUID="148ce8dc-1145-488d-940e-111d5cb3d9fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 00:43:54.031784 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:54.031236    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 00:43:54.031784 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:54.031312    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj" podUID="148ce8dc-1145-488d-940e-111d5cb3d9fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 00:43:55.027664 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:55.027624    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 00:43:55.028171 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:55.027712    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj" podUID="148ce8dc-1145-488d-940e-111d5cb3d9fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 00:43:56.032176 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:56.032145    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:56.032573 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:56.032551    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:43:56.033261 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:43:56.033241    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:44:03.588339 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.588301    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"]
Oct 31 00:44:03.593019 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.592994    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.600514 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.600462    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"]
Oct 31 00:44:03.604720 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.604699    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.619405 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.619377    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"]
Oct 31 00:44:03.647787 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.647757    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"]
Oct 31 00:44:03.713324 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.713286    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d509de-a07d-49ba-9a35-67873b87041b-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.713324 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.713326    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.713528 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.713343    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.713528 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.713444    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.713528 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.713502    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.713621 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.713598    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.713652 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.713626    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d509de-a07d-49ba-9a35-67873b87041b-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.713720 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.713698    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.713777 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.713762    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq8sr\" (UniqueName: \"kubernetes.io/projected/47d509de-a07d-49ba-9a35-67873b87041b-kube-api-access-wq8sr\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.713812 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.713795    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.713882 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.713871    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d509de-a07d-49ba-9a35-67873b87041b-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.713933 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.713922    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.713987 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.713976    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.714022 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.714012    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.714105 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.714094    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c58z2\" (UniqueName: \"kubernetes.io/projected/0260aaaa-f469-439d-91a1-45e0c4aab165-kube-api-access-c58z2\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.714152 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.714138    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0260aaaa-f469-439d-91a1-45e0c4aab165-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.714196 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.714181    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.714248 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.714234    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.815539 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.815501    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d509de-a07d-49ba-9a35-67873b87041b-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.815751 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.815558    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.815751 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.815608    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.815751 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.815647    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.815751 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.815737    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c58z2\" (UniqueName: \"kubernetes.io/projected/0260aaaa-f469-439d-91a1-45e0c4aab165-kube-api-access-c58z2\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.815951 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.815776    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0260aaaa-f469-439d-91a1-45e0c4aab165-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.815951 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.815814    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.816056 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.815997    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.816056 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.816049    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d509de-a07d-49ba-9a35-67873b87041b-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.816194 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.816152    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.816260 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.816201    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.816315 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.816295    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.816367 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.816329    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.816444 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.816421    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.816493 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.816467    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.816556 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.816489    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.816556 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.816495    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d509de-a07d-49ba-9a35-67873b87041b-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.816668 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.816558    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.816668 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.816624    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq8sr\" (UniqueName: \"kubernetes.io/projected/47d509de-a07d-49ba-9a35-67873b87041b-kube-api-access-wq8sr\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.816668 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.816660    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.816886 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.816852    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.816886 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.816874    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.817424 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.817262    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.817424 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.817325    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d509de-a07d-49ba-9a35-67873b87041b-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.817424 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.817380    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.817747 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.817443    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.817842 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.817815    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0260aaaa-f469-439d-91a1-45e0c4aab165-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.818002 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.817987    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.818561 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.818541    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.818844 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.818821    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d509de-a07d-49ba-9a35-67873b87041b-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.819081 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.819062    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.819363 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.819348    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.840224 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.840158    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.840355 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.840308    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq8sr\" (UniqueName: \"kubernetes.io/projected/47d509de-a07d-49ba-9a35-67873b87041b-kube-api-access-wq8sr\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.842078 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.842052    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d509de-a07d-49ba-9a35-67873b87041b-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:03.842202 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.842164    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c58z2\" (UniqueName: \"kubernetes.io/projected/0260aaaa-f469-439d-91a1-45e0c4aab165-kube-api-access-c58z2\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.909121 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.909090    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:03.918003 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:03.917937    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:04.249945 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:04.249918    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"]
Oct 31 00:44:04.250321 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:44:04.250290    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0260aaaa_f469_439d_91a1_45e0c4aab165.slice/crio-b096c64c581906da4c539073078095c0ff3350b64efe6630fb40ed72c58bd3f5 WatchSource:0}: Error finding container b096c64c581906da4c539073078095c0ff3350b64efe6630fb40ed72c58bd3f5: Status 404 returned error can't find the container with id b096c64c581906da4c539073078095c0ff3350b64efe6630fb40ed72c58bd3f5
Oct 31 00:44:04.253014 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:04.252978    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:44:04.253122 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:04.253065    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:44:04.253183 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:04.253123    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:44:04.260622 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:04.260599    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"]
Oct 31 00:44:04.271222 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:44:04.271188    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d509de_a07d_49ba_9a35_67873b87041b.slice/crio-543fd442e1bbe418a462a993288f7f0f888cf886b7a08b15cd3132b940df2f48 WatchSource:0}: Error finding container 543fd442e1bbe418a462a993288f7f0f888cf886b7a08b15cd3132b940df2f48: Status 404 returned error can't find the container with id 543fd442e1bbe418a462a993288f7f0f888cf886b7a08b15cd3132b940df2f48
Oct 31 00:44:04.274194 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:04.274156    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:44:04.274301 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:04.274253    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:44:04.274364 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:04.274303    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:44:04.699703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:04.687215    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x" event={"ID":"0260aaaa-f469-439d-91a1-45e0c4aab165","Type":"ContainerStarted","Data":"93d27ffdf13e5fa3cad8c5a67f1f02e8a9a5808c0bfecc5e7b8f829b5f3a3782"}
Oct 31 00:44:04.700349 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:04.700313    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x" event={"ID":"0260aaaa-f469-439d-91a1-45e0c4aab165","Type":"ContainerStarted","Data":"b096c64c581906da4c539073078095c0ff3350b64efe6630fb40ed72c58bd3f5"}
Oct 31 00:44:04.703983 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:04.703941    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj" event={"ID":"47d509de-a07d-49ba-9a35-67873b87041b","Type":"ContainerStarted","Data":"d783c099b11ddd7cc224d3fa18bc03971543d150ed3069f48f24a36d729b8b2a"}
Oct 31 00:44:04.704121 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:04.704027    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj" event={"ID":"47d509de-a07d-49ba-9a35-67873b87041b","Type":"ContainerStarted","Data":"543fd442e1bbe418a462a993288f7f0f888cf886b7a08b15cd3132b940df2f48"}
Oct 31 00:44:04.767559 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:04.767485    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x" podStartSLOduration=1.767465567 podStartE2EDuration="1.767465567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:04.763165722 +0000 UTC m=+1504.985201294" watchObservedRunningTime="2026-10-31 00:44:04.767465567 +0000 UTC m=+1504.989501108"
Oct 31 00:44:04.823373 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:04.823298    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj" podStartSLOduration=1.823276163 podStartE2EDuration="1.823276163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:04.816867758 +0000 UTC m=+1505.038903300" watchObservedRunningTime="2026-10-31 00:44:04.823276163 +0000 UTC m=+1505.045311705"
Oct 31 00:44:04.916712 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:04.913825    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:04.928746 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:04.924794    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:05.911700 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:05.911099    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 00:44:05.911700 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:05.911186    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x" podUID="0260aaaa-f469-439d-91a1-45e0c4aab165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 00:44:05.977706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:05.971891    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 00:44:05.977706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:05.971947    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj" podUID="47d509de-a07d-49ba-9a35-67873b87041b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 00:44:06.439916 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:06.436786    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:06.476175 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:06.452047    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:06.691758 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:06.688860    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 00:44:06.743842 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:06.743784    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:06.799902 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:06.799855    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 00:44:06.929095 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:06.928837    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 00:44:06.929095 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:06.928931    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj" podUID="47d509de-a07d-49ba-9a35-67873b87041b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 00:44:06.929095 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:06.928970    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 00:44:06.929095 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:06.929005    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x" podUID="0260aaaa-f469-439d-91a1-45e0c4aab165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 00:44:07.096672 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.083937    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:07.125707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.124770    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 00:44:07.283754 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.276806    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded"
Oct 31 00:44:07.480700 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.479773    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:44:07.480700 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.480576    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:44:07.529721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.528703    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"]
Oct 31 00:44:07.568827 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.567898    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:07.592116 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.592035    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d66e61c1-openshift-default-dockercfg-9hzdm\""
Oct 31 00:44:07.636504 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.636468    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"]
Oct 31 00:44:07.668749 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.668711    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-workload-socket\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:07.678938 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.678896    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-podinfo\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:07.679257 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.679237    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-data\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:07.679424 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.679409    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-envoy\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:07.679549 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.679536    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istiod-ca-cert\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:07.679748 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.679727    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-token\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:07.679993 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.679974    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-credential-socket\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:07.680094 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.680080    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-workload-certs\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:07.694701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.680460    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gb55\" (UniqueName: \"kubernetes.io/projected/71c6ffbe-9a30-4a28-aa62-57ecb4052308-kube-api-access-5gb55\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:07.695118 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.689364    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 00:44:07.697127 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.695755    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:44:07.751752 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.745857    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:07.751752 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.746594    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:44:07.813705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.798067    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gb55\" (UniqueName: \"kubernetes.io/projected/71c6ffbe-9a30-4a28-aa62-57ecb4052308-kube-api-access-5gb55\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:07.828716 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.823922    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-workload-socket\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:07.828716 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.824131    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-podinfo\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:07.828716 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.824272    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-data\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:07.828716 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.824367    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-envoy\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:07.828716 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.824406    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istiod-ca-cert\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:07.828716 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.824499    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-token\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:07.828716 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.824639    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-credential-socket\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:07.828716 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.824666    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-workload-certs\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:07.828716 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.825228    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-workload-certs\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:07.828716 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.813989    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:07.974704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.877826    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istiod-ca-cert\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:07.974704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.878417    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-credential-socket\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:07.974704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.878826    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:44:07.974704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:07.900798    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-envoy\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:08.119759 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:08.082232    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-token\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:08.182023 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:08.181494    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:08.182435 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:08.181929    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-workload-socket\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:08.187071 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:08.185750    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-data\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:08.187587 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:08.187564    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 00:44:08.187780 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:08.187754    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x" podUID="0260aaaa-f469-439d-91a1-45e0c4aab165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 00:44:08.308591 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:08.308367    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 00:44:08.308591 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:08.308426    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj" podUID="47d509de-a07d-49ba-9a35-67873b87041b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 00:44:08.399789 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:08.382845    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:08.491249 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:08.489964    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:44:08.740998 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:08.740892    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 00:44:08.763063 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:08.759043    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gb55\" (UniqueName: \"kubernetes.io/projected/71c6ffbe-9a30-4a28-aa62-57ecb4052308-kube-api-access-5gb55\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:08.788030 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:08.787977    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:08.926704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:08.883278    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:09.141135 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:08.963627    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-podinfo\") pod \"e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") " pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:09.198480 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:08.983887    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 00:44:09.198480 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:09.197225    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj" podUID="47d509de-a07d-49ba-9a35-67873b87041b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 00:44:09.198480 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:09.186768    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 00:44:09.198861 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:09.174863    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 00:44:09.198919 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:09.198878    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x" podUID="0260aaaa-f469-439d-91a1-45e0c4aab165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 00:44:09.205709 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:44:09.199560    4327 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.165s"
Oct 31 00:44:09.206022 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:09.205998    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:44:09.214967 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:09.214940    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:44:09.496423 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:09.496319    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:09.758796 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:09.496729    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:44:10.104795 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:09.538330    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:10.120170 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:10.120052    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 00:44:10.149161 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:09.813138    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:44:10.185705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:09.978520    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 00:44:10.185705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:09.990556    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 00:44:10.283704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:10.250917    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:10.283704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:10.251310    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:10.283704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:10.251383    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj" podUID="47d509de-a07d-49ba-9a35-67873b87041b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 00:44:10.283704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:10.251416    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x" podUID="0260aaaa-f469-439d-91a1-45e0c4aab165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 00:44:10.576762 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:09.813038    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:10.926741 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:10.919972    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 00:44:10.927054 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:10.927010    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj" podUID="47d509de-a07d-49ba-9a35-67873b87041b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 00:44:10.986707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:10.983266    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 00:44:11.152705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:11.150877    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x" podUID="0260aaaa-f469-439d-91a1-45e0c4aab165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 00:44:11.152705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:11.105904    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:11.152705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:11.098519    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:44:11.234902 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:11.234797    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:11.235471 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:11.235437    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 00:44:11.259854 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:11.259798    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:11.279343 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:11.278797    4327 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:44:11.279343 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:11.278866    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh" podUID="2a45ba96-dddd-43f5-9e50-0b08194799f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:11.577578 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:11.577477    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:11.769346 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:11.768845    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:44:11.769346 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:11.768938    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5" podUID="dbb151f9-f859-4faf-a830-9da6730837c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:11.952737 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:11.934256    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 00:44:11.952737 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:11.934313    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x" podUID="0260aaaa-f469-439d-91a1-45e0c4aab165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 00:44:11.952737 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:11.949595    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 00:44:11.952737 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:11.949651    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj" podUID="47d509de-a07d-49ba-9a35-67873b87041b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 00:44:11.990773 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:11.990726    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:12.012776 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:11.992855    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:44:12.012776 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:11.992922    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58" podUID="ed99fa70-689b-472e-8e25-62e70c3f2972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:12.211766 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:12.188313    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:12.211766 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:12.188663    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:44:12.237863 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:12.237801    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:12.254927 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:12.254875    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:12.275287 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:12.265258    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 00:44:12.361719 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:12.359782    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:44:12.361719 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:12.359863    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj" podUID="148ce8dc-1145-488d-940e-111d5cb3d9fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:12.372894 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:12.372852    4327 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:44:12.373178 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:12.373142    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd" podUID="49a4ff5e-3ec9-45ce-88c2-a9b6308a821e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:12.588819 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:12.584848    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 00:44:12.614079 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:12.593790    4327 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:44:12.614317 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:12.614286    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd" podUID="7b9657eb-33b5-4847-9ce7-39085e57f7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:12.934001 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:12.933669    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 00:44:12.934001 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:12.933753    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x" podUID="0260aaaa-f469-439d-91a1-45e0c4aab165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 00:44:12.953277 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:12.953240    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 00:44:12.953556 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:12.953525    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj" podUID="47d509de-a07d-49ba-9a35-67873b87041b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 00:44:13.005705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:12.997249    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"]
Oct 31 00:44:13.205988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:13.188464    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 00:44:13.205988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:13.189356    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:13.432713 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:13.424506    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:44:13.432713 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:13.424805    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 00:44:13.483765 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:13.454309    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:13.628706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:13.617814    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 00:44:13.658728 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:13.643585    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:44:13.675365 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:13.675303    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:44:13.683766 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:13.675665    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:44:14.176703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:14.166792    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt" podUID="c1398a6a-1ac9-48e1-a988-19c575e0cb91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:14.332219 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:14.303107    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 00:44:14.332219 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:14.303197    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x" podUID="0260aaaa-f469-439d-91a1-45e0c4aab165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 00:44:14.375008 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:14.356929    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:44:14.375008 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:14.357003    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx" podUID="02dc2389-ff84-4de4-a932-8be187b761e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:14.565337 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:14.462834    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 00:44:14.565337 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:14.470716    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:44:14.565337 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:14.544521    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:44:14.565337 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:14.545235    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 00:44:14.741825 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:14.722867    4327 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:44:14.741825 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:14.722947    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9" podUID="1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:15.392706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:15.385216    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 00:44:15.392706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:15.385301    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x" podUID="0260aaaa-f469-439d-91a1-45e0c4aab165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 00:44:15.416849 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:15.416234    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 00:44:15.416849 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:15.416565    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 00:44:15.482516 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:15.377671    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj" podUID="b1988c46-0b7f-4edd-b441-ab408ad0a246" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:15.482516 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:15.467928    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 00:44:15.482516 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:15.467994    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj" podUID="47d509de-a07d-49ba-9a35-67873b87041b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 00:44:15.482516 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:15.482023    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q" podUID="7c1e66f4-2552-43b1-96e0-272661898154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:15.482516 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:15.482153    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 00:44:15.488184 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:15.487926    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:15.584843 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:15.584807    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:15.586448 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:44:15.586364    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:44:15.589186 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:44:15.589158    4327 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.043s"
Oct 31 00:44:15.589294 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:15.589245    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:44:15.589334 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:15.589293    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:44:15.591868 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:15.591846    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:44:15.596844 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:15.596814    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq" event={"ID":"71c6ffbe-9a30-4a28-aa62-57ecb4052308","Type":"ContainerStarted","Data":"62af32dd6bfdc665c2b59530b8d89d3bca01ad2436fa866cd9729a28063ccdf9"}
Oct 31 00:44:15.603069 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:15.603036    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:15.604489 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:15.604467    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:44:15.611609 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:15.611584    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:44:15.683926 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:15.673210    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:44:15.910518 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:15.910478    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 00:44:15.910699 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:15.910538    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x" podUID="0260aaaa-f469-439d-91a1-45e0c4aab165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 00:44:15.988760 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:15.987477    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 00:44:15.988760 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:15.987535    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj" podUID="47d509de-a07d-49ba-9a35-67873b87041b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 00:44:16.605317 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:16.605281    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq" event={"ID":"71c6ffbe-9a30-4a28-aa62-57ecb4052308","Type":"ContainerStarted","Data":"ed985d03890e6cbe97135b8762d6b235f915f74f8855483228b907567d7d9584"}
Oct 31 00:44:16.672483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:16.672402    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq" podStartSLOduration=9.672388969 podStartE2EDuration="9.672388969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:16.670034699 +0000 UTC m=+1516.892070242" watchObservedRunningTime="2026-10-31 00:44:16.672388969 +0000 UTC m=+1516.894424510"
Oct 31 00:44:16.697021 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:16.696988    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:16.702597 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:16.702575    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:16.914941 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:16.911104    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 00:44:16.914941 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:16.911165    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x" podUID="0260aaaa-f469-439d-91a1-45e0c4aab165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 00:44:16.921270 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:16.921237    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 00:44:16.922133 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:16.922103    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj" podUID="47d509de-a07d-49ba-9a35-67873b87041b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 00:44:17.612501 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:17.612473    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:17.613620 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:17.613595    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:44:17.909846 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:17.909760    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 00:44:17.909846 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:17.909821    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x" podUID="0260aaaa-f469-439d-91a1-45e0c4aab165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 00:44:17.944704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:17.930315    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 00:44:17.945202 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:17.945167    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj" podUID="47d509de-a07d-49ba-9a35-67873b87041b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 00:44:18.909732 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:18.909664    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:18.910231 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:18.909915    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 00:44:18.910231 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:18.909982    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x" podUID="0260aaaa-f469-439d-91a1-45e0c4aab165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 00:44:18.918783 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:18.918749    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:18.919048 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:18.919023    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 00:44:18.919126 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:18.919075    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj" podUID="47d509de-a07d-49ba-9a35-67873b87041b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 00:44:19.920854 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:19.920818    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 00:44:19.921492 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:19.921437    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj" podUID="47d509de-a07d-49ba-9a35-67873b87041b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 00:44:20.685705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:20.642748    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6"]
Oct 31 00:44:20.685705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:20.648347    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6"
Oct 31 00:44:20.769914 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:20.725437    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d66e61c1\""
Oct 31 00:44:20.769914 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:20.726166    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d66e61c1-dockercfg-k8tgl\""
Oct 31 00:44:20.890563 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:20.858504    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6ede26-6ab9-4185-a5c2-793692a81e3d-plugins-config-volume\") pod \"payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6\" (UID: \"be6ede26-6ab9-4185-a5c2-793692a81e3d\") " pod="openshift-ingress/payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6"
Oct 31 00:44:20.890563 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:20.858579    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46fgv\" (UniqueName: \"kubernetes.io/projected/be6ede26-6ab9-4185-a5c2-793692a81e3d-kube-api-access-46fgv\") pod \"payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6\" (UID: \"be6ede26-6ab9-4185-a5c2-793692a81e3d\") " pod="openshift-ingress/payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6"
Oct 31 00:44:20.958820 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:20.956602    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6"]
Oct 31 00:44:21.036860 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:21.021326    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6ede26-6ab9-4185-a5c2-793692a81e3d-plugins-config-volume\") pod \"payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6\" (UID: \"be6ede26-6ab9-4185-a5c2-793692a81e3d\") " pod="openshift-ingress/payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6"
Oct 31 00:44:21.036860 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:21.021387    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46fgv\" (UniqueName: \"kubernetes.io/projected/be6ede26-6ab9-4185-a5c2-793692a81e3d-kube-api-access-46fgv\") pod \"payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6\" (UID: \"be6ede26-6ab9-4185-a5c2-793692a81e3d\") " pod="openshift-ingress/payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6"
Oct 31 00:44:21.036860 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:21.031797    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:44:21.036860 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:21.031891    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x" podUID="0260aaaa-f469-439d-91a1-45e0c4aab165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:21.059474 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:21.059439    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6ede26-6ab9-4185-a5c2-793692a81e3d-plugins-config-volume\") pod \"payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6\" (UID: \"be6ede26-6ab9-4185-a5c2-793692a81e3d\") " pod="openshift-ingress/payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6"
Oct 31 00:44:21.129748 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:21.122311    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46fgv\" (UniqueName: \"kubernetes.io/projected/be6ede26-6ab9-4185-a5c2-793692a81e3d-kube-api-access-46fgv\") pod \"payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6\" (UID: \"be6ede26-6ab9-4185-a5c2-793692a81e3d\") " pod="openshift-ingress/payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6"
Oct 31 00:44:21.383494 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:21.383451    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6"
Oct 31 00:44:21.776759 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:21.768420    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6"]
Oct 31 00:44:21.803422 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:44:21.801069    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe6ede26_6ab9_4185_a5c2_793692a81e3d.slice/crio-5d10ebfc6be0ba7b98bc5759461611d079f8e182d4727eb7d5be66f65c53aa24 WatchSource:0}: Error finding container 5d10ebfc6be0ba7b98bc5759461611d079f8e182d4727eb7d5be66f65c53aa24: Status 404 returned error can't find the container with id 5d10ebfc6be0ba7b98bc5759461611d079f8e182d4727eb7d5be66f65c53aa24
Oct 31 00:44:22.032762 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:22.031240    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:44:22.032762 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:22.031308    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj" podUID="47d509de-a07d-49ba-9a35-67873b87041b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:22.090798 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:22.090756    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:44:22.090992 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:22.090816    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x" podUID="0260aaaa-f469-439d-91a1-45e0c4aab165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:22.092499 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:22.092460    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6" event={"ID":"be6ede26-6ab9-4185-a5c2-793692a81e3d","Type":"ContainerStarted","Data":"3d3bee6ecfd3fb23a8c1476392619c7b4ca87065a1a744626cad0ec91ed901df"}
Oct 31 00:44:22.092606 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:22.092532    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6" event={"ID":"be6ede26-6ab9-4185-a5c2-793692a81e3d","Type":"ContainerStarted","Data":"5d10ebfc6be0ba7b98bc5759461611d079f8e182d4727eb7d5be66f65c53aa24"}
Oct 31 00:44:22.092822 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:22.092786    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6"
Oct 31 00:44:22.135387 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:22.135324    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6" podStartSLOduration=2.135304707 podStartE2EDuration="2.135304707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:22.135109835 +0000 UTC m=+1522.357145380" watchObservedRunningTime="2026-10-31 00:44:22.135304707 +0000 UTC m=+1522.357340251"
Oct 31 00:44:22.617479 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:22.617404    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:22.620080 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:22.618913    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:44:22.776711 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:22.776267    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:22.778104 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:22.778077    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:44:28.591145 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.591104    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"]
Oct 31 00:44:28.643905 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.643875    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"]
Oct 31 00:44:28.644176 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.644163    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.665373 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.665347    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-f598cd05-gw-openshift-default-dockercfg-fm8l4\""
Oct 31 00:44:28.773536 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.773487    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4td7l\" (UniqueName: \"kubernetes.io/projected/87c3205a-74bd-42f8-98cc-114cf902ed9d-kube-api-access-4td7l\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.784816 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.784781    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-token\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.785095 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.785079    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-data\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.790773 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.785281    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-envoy\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.791296 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.791266    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-workload-socket\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.795856 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.791652    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-credential-socket\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.795856 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.791752    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c3205a-74bd-42f8-98cc-114cf902ed9d-istiod-ca-cert\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.795856 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.791863    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-workload-certs\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.795856 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.791952    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-podinfo\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.909028 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.902024    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-token\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.909028 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.902075    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-data\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.909028 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.902128    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-envoy\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.909028 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.902216    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-workload-socket\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.909028 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.902288    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-credential-socket\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.909028 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.902324    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c3205a-74bd-42f8-98cc-114cf902ed9d-istiod-ca-cert\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.909028 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.902356    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-workload-certs\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.909028 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.902394    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-podinfo\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.909028 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.902549    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4td7l\" (UniqueName: \"kubernetes.io/projected/87c3205a-74bd-42f8-98cc-114cf902ed9d-kube-api-access-4td7l\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.909028 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.903333    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-data\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.909028 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.905998    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-envoy\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.909028 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.906211    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-workload-socket\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.909028 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.906377    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-credential-socket\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.909028 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.907059    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c3205a-74bd-42f8-98cc-114cf902ed9d-istiod-ca-cert\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.909028 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.907369    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-workload-certs\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.916742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.911816    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-podinfo\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.930110 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.930078    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-token\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.930279 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.930258    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4td7l\" (UniqueName: \"kubernetes.io/projected/87c3205a-74bd-42f8-98cc-114cf902ed9d-kube-api-access-4td7l\") pod \"e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") " pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:28.964926 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:28.964891    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:29.414708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:29.414657    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"]
Oct 31 00:44:29.417902 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:44:29.417872    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c3205a_74bd_42f8_98cc_114cf902ed9d.slice/crio-c7d034c582cac510996f151999be31b82a1455b4b1bd0a208a57ea9e70fab092 WatchSource:0}: Error finding container c7d034c582cac510996f151999be31b82a1455b4b1bd0a208a57ea9e70fab092: Status 404 returned error can't find the container with id c7d034c582cac510996f151999be31b82a1455b4b1bd0a208a57ea9e70fab092
Oct 31 00:44:29.421428 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:29.421376    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:44:29.421552 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:29.421528    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:44:29.421640 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:29.421618    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:44:29.814541 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:29.814456    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm"]
Oct 31 00:44:29.819538 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:29.819507    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm"
Oct 31 00:44:29.875146 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:29.875016    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a9461dc0\""
Oct 31 00:44:29.875536 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:29.875514    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a9461dc0-dockercfg-7s76n\""
Oct 31 00:44:30.035477 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:30.035432    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b1f2ea-dbf5-44de-adeb-e763db1ee2ce-plugins-config-volume\") pod \"payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm\" (UID: \"06b1f2ea-dbf5-44de-adeb-e763db1ee2ce\") " pod="openshift-ingress/payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm"
Oct 31 00:44:30.035878 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:30.035849    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcbwt\" (UniqueName: \"kubernetes.io/projected/06b1f2ea-dbf5-44de-adeb-e763db1ee2ce-kube-api-access-kcbwt\") pod \"payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm\" (UID: \"06b1f2ea-dbf5-44de-adeb-e763db1ee2ce\") " pod="openshift-ingress/payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm"
Oct 31 00:44:30.139125 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:30.139088    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b1f2ea-dbf5-44de-adeb-e763db1ee2ce-plugins-config-volume\") pod \"payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm\" (UID: \"06b1f2ea-dbf5-44de-adeb-e763db1ee2ce\") " pod="openshift-ingress/payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm"
Oct 31 00:44:30.139609 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:30.139558    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcbwt\" (UniqueName: \"kubernetes.io/projected/06b1f2ea-dbf5-44de-adeb-e763db1ee2ce-kube-api-access-kcbwt\") pod \"payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm\" (UID: \"06b1f2ea-dbf5-44de-adeb-e763db1ee2ce\") " pod="openshift-ingress/payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm"
Oct 31 00:44:30.143412 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:30.141234    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b1f2ea-dbf5-44de-adeb-e763db1ee2ce-plugins-config-volume\") pod \"payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm\" (UID: \"06b1f2ea-dbf5-44de-adeb-e763db1ee2ce\") " pod="openshift-ingress/payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm"
Oct 31 00:44:30.178479 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:30.178443    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcbwt\" (UniqueName: \"kubernetes.io/projected/06b1f2ea-dbf5-44de-adeb-e763db1ee2ce-kube-api-access-kcbwt\") pod \"payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm\" (UID: \"06b1f2ea-dbf5-44de-adeb-e763db1ee2ce\") " pod="openshift-ingress/payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm"
Oct 31 00:44:30.183774 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:30.182336    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm"]
Oct 31 00:44:30.246704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:30.240822    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm"
Oct 31 00:44:30.347589 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:30.347556    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78" event={"ID":"87c3205a-74bd-42f8-98cc-114cf902ed9d","Type":"ContainerStarted","Data":"08522a0fe2d8b3c408453001fe6846509f241eee7c6b7beb8cec67b9ff942654"}
Oct 31 00:44:30.347842 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:30.347789    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78" event={"ID":"87c3205a-74bd-42f8-98cc-114cf902ed9d","Type":"ContainerStarted","Data":"c7d034c582cac510996f151999be31b82a1455b4b1bd0a208a57ea9e70fab092"}
Oct 31 00:44:30.407212 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:30.407104    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78" podStartSLOduration=2.407086861 podStartE2EDuration="2.407086861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:30.402780291 +0000 UTC m=+1530.624815859" watchObservedRunningTime="2026-10-31 00:44:30.407086861 +0000 UTC m=+1530.629122404"
Oct 31 00:44:30.714522 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:30.714342    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm"]
Oct 31 00:44:30.965425 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:30.965354    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:30.972004 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:30.971973    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:31.357105 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:31.357063    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm" event={"ID":"06b1f2ea-dbf5-44de-adeb-e763db1ee2ce","Type":"ContainerStarted","Data":"b3214075827d05add29b445423c2187f0c9f7c26fa62fcbc40f69c0bf217a1a9"}
Oct 31 00:44:31.357294 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:31.357131    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm" event={"ID":"06b1f2ea-dbf5-44de-adeb-e763db1ee2ce","Type":"ContainerStarted","Data":"491801eae2fe214f6c99c9280242350a5b954c29fb498d8ccf4e6d2c1876c736"}
Oct 31 00:44:31.357294 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:31.357164    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm"
Oct 31 00:44:31.357438 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:31.357289    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:31.358244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:31.358226    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:44:31.401849 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:31.401798    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm" podStartSLOduration=2.401781273 podStartE2EDuration="2.401781273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:31.397214698 +0000 UTC m=+1531.619250240" watchObservedRunningTime="2026-10-31 00:44:31.401781273 +0000 UTC m=+1531.623816815"
Oct 31 00:44:33.103601 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:33.103574    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6"
Oct 31 00:44:39.813792 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:39.813674    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"]
Oct 31 00:44:39.824484 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:39.824453    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:39.879520 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:39.876531    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b08f569a-openshift-default-dockercfg-vnlqs\""
Oct 31 00:44:39.936733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:39.935830    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istiod-ca-cert\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:39.936733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:39.935948    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-podinfo\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:39.936733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:39.936011    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-token\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:39.936733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:39.936071    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-credential-socket\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:39.936733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:39.936109    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-workload-certs\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:39.936733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:39.936228    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-envoy\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:39.936733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:39.936313    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-workload-socket\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:39.936733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:39.936363    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-data\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:39.936733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:39.936414    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkr4r\" (UniqueName: \"kubernetes.io/projected/ff2dc9c9-da34-4576-b759-8f1f6b632e51-kube-api-access-lkr4r\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:39.958458 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:39.958414    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"]
Oct 31 00:44:40.038068 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:40.038021    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istiod-ca-cert\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:40.038250 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:40.038130    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-podinfo\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:40.038250 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:40.038169    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-token\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:40.038250 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:40.038215    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-credential-socket\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:40.038250 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:40.038238    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-workload-certs\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:40.038459 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:40.038340    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-envoy\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:40.038459 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:40.038412    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-workload-socket\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:40.038573 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:40.038458    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-data\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:40.038573 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:40.038502    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkr4r\" (UniqueName: \"kubernetes.io/projected/ff2dc9c9-da34-4576-b759-8f1f6b632e51-kube-api-access-lkr4r\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:40.038735 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:40.038707    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-workload-certs\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:40.038916 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:40.038894    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-credential-socket\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:40.038966 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:40.038903    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-data\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:40.039006 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:40.038956    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-workload-socket\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:40.039376 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:40.039357    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istiod-ca-cert\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:40.040873 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:40.040851    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-envoy\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:40.041643 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:40.041616    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-podinfo\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:40.062128 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:40.062092    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-token\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:40.062280 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:40.062191    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkr4r\" (UniqueName: \"kubernetes.io/projected/ff2dc9c9-da34-4576-b759-8f1f6b632e51-kube-api-access-lkr4r\") pod \"e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") " pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:40.260706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:40.257997    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:41.157812 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:41.157765    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:41.220791 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:44:41.220758    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff2dc9c9_da34_4576_b759_8f1f6b632e51.slice/crio-196dfc374197b206a540a69f96d3b81ddefd6d5a9fd6ae96402c9e5b79450cac WatchSource:0}: Error finding container 196dfc374197b206a540a69f96d3b81ddefd6d5a9fd6ae96402c9e5b79450cac: Status 404 returned error can't find the container with id 196dfc374197b206a540a69f96d3b81ddefd6d5a9fd6ae96402c9e5b79450cac
Oct 31 00:44:41.229941 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:41.229871    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"]
Oct 31 00:44:41.247107 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:41.247062    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:44:41.247391 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:41.247364    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:44:41.247566 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:41.247546    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:44:41.516257 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:41.516153    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd" event={"ID":"ff2dc9c9-da34-4576-b759-8f1f6b632e51","Type":"ContainerStarted","Data":"196dfc374197b206a540a69f96d3b81ddefd6d5a9fd6ae96402c9e5b79450cac"}
Oct 31 00:44:42.412945 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:42.412906    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm"
Oct 31 00:44:42.814765 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:42.778463    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:42.814765 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:42.808156    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd" event={"ID":"ff2dc9c9-da34-4576-b759-8f1f6b632e51","Type":"ContainerStarted","Data":"4eb25bf775888fe94a95f96a52dfd4a8037da2ab69019898c857f66ca9dbb23d"}
Oct 31 00:44:43.203875 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:43.203206    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:43.255707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:43.249521    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:43.255707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:43.249870    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 00:44:43.257811 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:43.257066    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:43.281710 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:43.281469    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:43.281710 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:43.281622    4327 patch_prober.go:29] interesting pod/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 00:44:43.293857 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:43.281670    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd" podUID="ff2dc9c9-da34-4576-b759-8f1f6b632e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 00:44:43.508863 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:43.507776    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:44:43.868705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:43.867419    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:43.868705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:43.868032    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:44:44.247945 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:44.238101    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 00:44:44.312704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:44.311891    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 00:44:44.413704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:44.396045    4327 patch_prober.go:29] interesting pod/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 00:44:44.413704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:44.396107    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd" podUID="ff2dc9c9-da34-4576-b759-8f1f6b632e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 00:44:44.441707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:44.440622    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 00:44:44.652704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:44.651901    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:44:44.699708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:44.696636    4327 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:44:44.699708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:44.696745    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55" podUID="93505d53-b42a-4c07-ae52-d62ac15f1d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:44.811741 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:44.811674    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:44:44.906405 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:44.882051    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:45.092285 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:45.092251    4327 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:44:45.323475 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:45.323358    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp" podUID="f32926f1-4427-4764-b86d-7294eaba9e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:45.323887 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:45.323853    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:45.325334 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:45.108813    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd" podStartSLOduration=6.108789295 podStartE2EDuration="6.108789295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:42.873240515 +0000 UTC m=+1543.095276054" watchObservedRunningTime="2026-10-31 00:44:45.108789295 +0000 UTC m=+1545.330824835"
Oct 31 00:44:45.325585 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:44:45.284952    4327 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.324s"
Oct 31 00:44:45.344872 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:45.344829    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:44:45.345190 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:45.345171    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:44:45.348811 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:45.348781    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:44:45.375702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:45.349037    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:44:45.376029 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:45.376005    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:44:45.477706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:45.243927    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:45.638706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:45.554049    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:45.654730 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:45.653407    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:45.714705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:45.710654    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded"
Oct 31 00:44:45.850738 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:45.827246    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:44:45.899835 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:45.889788    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:46.350991 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:46.345761    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 00:44:46.491227 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:46.491185    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:44:46.491558 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:46.491505    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg" podUID="2bd889fe-02e5-45e4-aaa6-156d3beabf00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:46.605758 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:46.605640    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded"
Oct 31 00:44:46.783866 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:46.783796    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:46.798821 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:46.798770    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:44:46.805985 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:46.805942    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:46.901049 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:46.900951    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:44:46.944322 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:46.901319    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:46.972659 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:46.970480    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:44:46.972659 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:46.970543    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh" podUID="b1acb532-b591-421f-87a7-61fbaec51ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:47.122103 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:44:47.122057    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:44:47.428769 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:44:47.377500    4327 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.434s"
Oct 31 00:44:47.428769 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:47.377547    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:47.428769 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:47.377598    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:47.428769 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:47.391116    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:44:47.483223 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:47.483182    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 00:44:47.557407 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:47.557347    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:47.793763 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:47.785115    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 00:44:47.828825 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:47.828777    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 00:44:47.851703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:47.828996    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:47.993706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:47.952962    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:48.114761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:48.080042    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:44:48.540207 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:48.540077    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 00:44:48.841165 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:48.840783    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 00:44:48.842030 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:48.841979    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:48.865903 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:48.865848    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:44:48.989234 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:48.989182    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 00:44:49.055704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:49.054928    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:44:49.449713 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:49.446380    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:49.744755 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:49.736850    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 00:44:49.842887 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:49.842835    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:49.844165 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:49.844130    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:49.873574 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:49.873527    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:50.038650 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:50.037381    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:50.150703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:50.104761    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:50.206045 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:50.196649    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:50.737965 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:50.737914    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:51.028772 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:51.027936    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 00:44:51.028772 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:51.028500    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 00:44:51.080710 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:51.042230    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 00:44:51.080710 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:51.051093    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:51.113856 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:51.113808    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:44:51.320087 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:44:51.319993    4327 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.401s"
Oct 31 00:44:51.320450 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:51.320415    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:44:51.786207 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:51.786159    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:51.835199 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:51.803497    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:44:52.067770 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:52.053992    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:52.067770 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:52.054464    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded"
Oct 31 00:44:52.149340 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:52.138477    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:52.149340 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:52.141149    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:52.149340 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:52.146453    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:44:52.216736 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:52.198530    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hk9qf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:44:52.287705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:52.284181    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf" podUID="a5f7f289-ef35-494d-91c9-034c7f1294d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:52.292244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:52.198555    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4qhrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:44:52.292482 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:52.292452    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd" podUID="6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:52.644664 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:52.644629    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:44:52.810847 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:52.810797    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:52.842702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:52.836764    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:52.869629 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:52.869599    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:44:52.981097 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:52.978283    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:44:53.224698 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:53.202864    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:44:53.224698 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:53.203328    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:44:53.426907 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:53.407844    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:44:53.442172 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:44:53.442146    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:45:00.238268 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:00.238224    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf"]
Oct 31 00:45:00.242904 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:00.242880    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf"
Oct 31 00:45:00.277367 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:00.277054    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf"]
Oct 31 00:45:00.278929 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:00.278893    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d477l\" (UniqueName: \"kubernetes.io/projected/e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f-kube-api-access-d477l\") pod \"maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf\" (UID: \"e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf"
Oct 31 00:45:00.279116 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:00.278950    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf\" (UID: \"e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf"
Oct 31 00:45:00.379466 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:00.356551    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\""
Oct 31 00:45:00.419987 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:00.419933    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d477l\" (UniqueName: \"kubernetes.io/projected/e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f-kube-api-access-d477l\") pod \"maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf\" (UID: \"e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf"
Oct 31 00:45:00.420256 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:00.420230    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf\" (UID: \"e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf"
Oct 31 00:45:00.421456 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:00.421243    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf\" (UID: \"e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf"
Oct 31 00:45:00.478977 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:00.478942    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d477l\" (UniqueName: \"kubernetes.io/projected/e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f-kube-api-access-d477l\") pod \"maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf\" (UID: \"e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf"
Oct 31 00:45:00.557988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:00.557898    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf"
Oct 31 00:45:00.880783 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:00.880757    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf"]
Oct 31 00:45:00.881948 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:45:00.881921    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17ec3d6_2c2c_421b_a5c0_93692ca0ae4f.slice/crio-e3a1ff2362a1af4d7ac3e59c4898f7a47dd2fc0f030758aa0e67a35956ac717e WatchSource:0}: Error finding container e3a1ff2362a1af4d7ac3e59c4898f7a47dd2fc0f030758aa0e67a35956ac717e: Status 404 returned error can't find the container with id e3a1ff2362a1af4d7ac3e59c4898f7a47dd2fc0f030758aa0e67a35956ac717e
Oct 31 00:45:00.884132 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:00.884115    4327 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:45:01.156750 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:01.151024    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf" event={"ID":"e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f","Type":"ContainerStarted","Data":"e3a1ff2362a1af4d7ac3e59c4898f7a47dd2fc0f030758aa0e67a35956ac717e"}
Oct 31 00:45:05.196760 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:05.196720    4327 generic.go:356] "Generic (PLEG): container finished" podID="e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f" containerID="a61584f26aa239ed98c81c57351b42e060ca3008a96b8affd16c132b6a4f706d" exitCode=60
Oct 31 00:45:05.197165 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:05.196832    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf" event={"ID":"e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f","Type":"ContainerDied","Data":"a61584f26aa239ed98c81c57351b42e060ca3008a96b8affd16c132b6a4f706d"}
Oct 31 00:45:05.197232 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:05.197217    4327 scope.go:122] "RemoveContainer" containerID="a61584f26aa239ed98c81c57351b42e060ca3008a96b8affd16c132b6a4f706d"
Oct 31 00:45:06.207071 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:06.207031    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf" event={"ID":"e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f","Type":"ContainerStarted","Data":"c65ee404405e6a48ca1528fd7cf1041ca7df374854b250894944cf6b9cfb9837"}
Oct 31 00:45:06.257091 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:06.257033    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf" podStartSLOduration=3.429510773 podStartE2EDuration="6.257017838s" totalImagesPullingTime="2.827507065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:06.252737439 +0000 UTC m=+1566.474772984" watchObservedRunningTime="2026-10-31 00:45:06.257017838 +0000 UTC m=+1566.479053379"
Oct 31 00:45:07.219603 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:07.219556    4327 generic.go:356] "Generic (PLEG): container finished" podID="e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f" containerID="c65ee404405e6a48ca1528fd7cf1041ca7df374854b250894944cf6b9cfb9837" exitCode=60
Oct 31 00:45:07.220723 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:07.220155    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf" event={"ID":"e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f","Type":"ContainerDied","Data":"c65ee404405e6a48ca1528fd7cf1041ca7df374854b250894944cf6b9cfb9837"}
Oct 31 00:45:07.220866 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:07.220595    4327 scope.go:122] "RemoveContainer" containerID="c65ee404405e6a48ca1528fd7cf1041ca7df374854b250894944cf6b9cfb9837"
Oct 31 00:45:07.221096 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:07.221083    4327 scope.go:122] "RemoveContainer" containerID="a61584f26aa239ed98c81c57351b42e060ca3008a96b8affd16c132b6a4f706d"
Oct 31 00:45:07.221282 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:45:07.221252    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf_odh-ai-gateway-infra(e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf" podUID="e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f"
Oct 31 00:45:08.232563 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:08.232537    4327 scope.go:122] "RemoveContainer" containerID="c65ee404405e6a48ca1528fd7cf1041ca7df374854b250894944cf6b9cfb9837"
Oct 31 00:45:08.233055 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:45:08.233026    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf_odh-ai-gateway-infra(e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf" podUID="e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f"
Oct 31 00:45:10.701623 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:10.701585    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws"]
Oct 31 00:45:10.705842 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:10.705820    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws"
Oct 31 00:45:10.721402 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:10.721373    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:45:10.728870 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:10.728830    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws"]
Oct 31 00:45:10.790024 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:10.789976    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6cf9baa-9f6f-450f-be46-557469f45fde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws\" (UID: \"a6cf9baa-9f6f-450f-be46-557469f45fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws"
Oct 31 00:45:10.790310 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:10.790128    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxrrq\" (UniqueName: \"kubernetes.io/projected/a6cf9baa-9f6f-450f-be46-557469f45fde-kube-api-access-sxrrq\") pod \"maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws\" (UID: \"a6cf9baa-9f6f-450f-be46-557469f45fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws"
Oct 31 00:45:10.790476 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:10.790454    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6cf9baa-9f6f-450f-be46-557469f45fde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws\" (UID: \"a6cf9baa-9f6f-450f-be46-557469f45fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws"
Oct 31 00:45:10.892125 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:10.892086    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6cf9baa-9f6f-450f-be46-557469f45fde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws\" (UID: \"a6cf9baa-9f6f-450f-be46-557469f45fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws"
Oct 31 00:45:10.892452 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:10.892215    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6cf9baa-9f6f-450f-be46-557469f45fde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws\" (UID: \"a6cf9baa-9f6f-450f-be46-557469f45fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws"
Oct 31 00:45:10.892452 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:10.892276    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxrrq\" (UniqueName: \"kubernetes.io/projected/a6cf9baa-9f6f-450f-be46-557469f45fde-kube-api-access-sxrrq\") pod \"maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws\" (UID: \"a6cf9baa-9f6f-450f-be46-557469f45fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws"
Oct 31 00:45:10.892810 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:10.892789    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6cf9baa-9f6f-450f-be46-557469f45fde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws\" (UID: \"a6cf9baa-9f6f-450f-be46-557469f45fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws"
Oct 31 00:45:10.894775 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:10.894752    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6cf9baa-9f6f-450f-be46-557469f45fde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws\" (UID: \"a6cf9baa-9f6f-450f-be46-557469f45fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws"
Oct 31 00:45:10.918386 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:10.918357    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxrrq\" (UniqueName: \"kubernetes.io/projected/a6cf9baa-9f6f-450f-be46-557469f45fde-kube-api-access-sxrrq\") pod \"maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws\" (UID: \"a6cf9baa-9f6f-450f-be46-557469f45fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws"
Oct 31 00:45:10.942009 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:10.941978    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg"]
Oct 31 00:45:10.946590 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:10.946560    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg"
Oct 31 00:45:10.968119 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:10.966965    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b08f569a-serving-cert\""
Oct 31 00:45:10.970101 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:10.969599    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg"]
Oct 31 00:45:11.019654 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:11.019622    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws"
Oct 31 00:45:11.101834 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:11.101801    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgf2s\" (UniqueName: \"kubernetes.io/projected/191c12a0-fb78-4586-8d87-80f3cf3dce22-kube-api-access-jgf2s\") pod \"maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg\" (UID: \"191c12a0-fb78-4586-8d87-80f3cf3dce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg"
Oct 31 00:45:11.101986 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:11.101922    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/191c12a0-fb78-4586-8d87-80f3cf3dce22-maas-api-tls\") pod \"maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg\" (UID: \"191c12a0-fb78-4586-8d87-80f3cf3dce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg"
Oct 31 00:45:11.102271 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:11.101986    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/191c12a0-fb78-4586-8d87-80f3cf3dce22-openshift-service-ca\") pod \"maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg\" (UID: \"191c12a0-fb78-4586-8d87-80f3cf3dce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg"
Oct 31 00:45:11.102271 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:11.102136    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/191c12a0-fb78-4586-8d87-80f3cf3dce22-metrics-certs\") pod \"maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg\" (UID: \"191c12a0-fb78-4586-8d87-80f3cf3dce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg"
Oct 31 00:45:11.209040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:11.205933    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/191c12a0-fb78-4586-8d87-80f3cf3dce22-metrics-certs\") pod \"maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg\" (UID: \"191c12a0-fb78-4586-8d87-80f3cf3dce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg"
Oct 31 00:45:11.209040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:11.206246    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgf2s\" (UniqueName: \"kubernetes.io/projected/191c12a0-fb78-4586-8d87-80f3cf3dce22-kube-api-access-jgf2s\") pod \"maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg\" (UID: \"191c12a0-fb78-4586-8d87-80f3cf3dce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg"
Oct 31 00:45:11.209040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:11.206352    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/191c12a0-fb78-4586-8d87-80f3cf3dce22-maas-api-tls\") pod \"maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg\" (UID: \"191c12a0-fb78-4586-8d87-80f3cf3dce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg"
Oct 31 00:45:11.209040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:11.206414    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/191c12a0-fb78-4586-8d87-80f3cf3dce22-openshift-service-ca\") pod \"maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg\" (UID: \"191c12a0-fb78-4586-8d87-80f3cf3dce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg"
Oct 31 00:45:11.209040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:11.207523    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/191c12a0-fb78-4586-8d87-80f3cf3dce22-openshift-service-ca\") pod \"maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg\" (UID: \"191c12a0-fb78-4586-8d87-80f3cf3dce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg"
Oct 31 00:45:11.209040 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:45:11.208614    4327 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-serving-cert: secret "maas-api-e2e-ait-b08f569a-serving-cert" not found
Oct 31 00:45:11.209040 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:45:11.208698    4327 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/191c12a0-fb78-4586-8d87-80f3cf3dce22-maas-api-tls podName:191c12a0-fb78-4586-8d87-80f3cf3dce22 nodeName:}" failed. No retries permitted until 2026-10-31 00:45:11.708661947 +0000 UTC m=+1571.930697481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/191c12a0-fb78-4586-8d87-80f3cf3dce22-maas-api-tls") pod "maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg" (UID: "191c12a0-fb78-4586-8d87-80f3cf3dce22") : secret "maas-api-e2e-ait-b08f569a-serving-cert" not found
Oct 31 00:45:11.261702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:11.233654    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/191c12a0-fb78-4586-8d87-80f3cf3dce22-metrics-certs\") pod \"maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg\" (UID: \"191c12a0-fb78-4586-8d87-80f3cf3dce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg"
Oct 31 00:45:11.261702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:11.237526    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgf2s\" (UniqueName: \"kubernetes.io/projected/191c12a0-fb78-4586-8d87-80f3cf3dce22-kube-api-access-jgf2s\") pod \"maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg\" (UID: \"191c12a0-fb78-4586-8d87-80f3cf3dce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg"
Oct 31 00:45:11.539708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:11.539660    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws"]
Oct 31 00:45:11.540084 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:45:11.540058    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6cf9baa_9f6f_450f_be46_557469f45fde.slice/crio-d515c5555e3a0f03905a63582411f7edbcc003fb9a0906465358a81ac75e747e WatchSource:0}: Error finding container d515c5555e3a0f03905a63582411f7edbcc003fb9a0906465358a81ac75e747e: Status 404 returned error can't find the container with id d515c5555e3a0f03905a63582411f7edbcc003fb9a0906465358a81ac75e747e
Oct 31 00:45:11.721349 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:11.721319    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/191c12a0-fb78-4586-8d87-80f3cf3dce22-maas-api-tls\") pod \"maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg\" (UID: \"191c12a0-fb78-4586-8d87-80f3cf3dce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg"
Oct 31 00:45:11.723749 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:11.723730    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/191c12a0-fb78-4586-8d87-80f3cf3dce22-maas-api-tls\") pod \"maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg\" (UID: \"191c12a0-fb78-4586-8d87-80f3cf3dce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg"
Oct 31 00:45:11.861866 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:11.861834    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg"
Oct 31 00:45:12.194645 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:45:12.194602    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191c12a0_fb78_4586_8d87_80f3cf3dce22.slice/crio-a188901a0645c752f0d600600e77dac7bf6217363d3aad9d7a6d21ed21facda0 WatchSource:0}: Error finding container a188901a0645c752f0d600600e77dac7bf6217363d3aad9d7a6d21ed21facda0: Status 404 returned error can't find the container with id a188901a0645c752f0d600600e77dac7bf6217363d3aad9d7a6d21ed21facda0
Oct 31 00:45:12.195055 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:12.195024    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg"]
Oct 31 00:45:12.391062 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:12.390232    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws" event={"ID":"a6cf9baa-9f6f-450f-be46-557469f45fde","Type":"ContainerStarted","Data":"525ba6907ee5358be6ec8b5b931b680e709ed66071c9f7eca1d88321b77343f2"}
Oct 31 00:45:12.391062 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:12.390298    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws" event={"ID":"a6cf9baa-9f6f-450f-be46-557469f45fde","Type":"ContainerStarted","Data":"d515c5555e3a0f03905a63582411f7edbcc003fb9a0906465358a81ac75e747e"}
Oct 31 00:45:12.397628 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:12.392440    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg" event={"ID":"191c12a0-fb78-4586-8d87-80f3cf3dce22","Type":"ContainerStarted","Data":"a188901a0645c752f0d600600e77dac7bf6217363d3aad9d7a6d21ed21facda0"}
Oct 31 00:45:12.546688 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:12.546569    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws" podStartSLOduration=2.546547374 podStartE2EDuration="2.546547374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:12.519391713 +0000 UTC m=+1572.741427260" watchObservedRunningTime="2026-10-31 00:45:12.546547374 +0000 UTC m=+1572.768582917"
Oct 31 00:45:13.406270 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:13.406231    4327 generic.go:356] "Generic (PLEG): container finished" podID="a6cf9baa-9f6f-450f-be46-557469f45fde" containerID="525ba6907ee5358be6ec8b5b931b680e709ed66071c9f7eca1d88321b77343f2" exitCode=0
Oct 31 00:45:13.407091 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:13.407062    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws" event={"ID":"a6cf9baa-9f6f-450f-be46-557469f45fde","Type":"ContainerDied","Data":"525ba6907ee5358be6ec8b5b931b680e709ed66071c9f7eca1d88321b77343f2"}
Oct 31 00:45:13.412599 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:13.412565    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg" event={"ID":"191c12a0-fb78-4586-8d87-80f3cf3dce22","Type":"ContainerStarted","Data":"3311a8a035a6f8553a080ac0db77b89b5e167b96ac82c9f022ac9e400979003a"}
Oct 31 00:45:13.414141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:13.414068    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg"
Oct 31 00:45:13.495754 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:13.495670    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg" podStartSLOduration=3.122314621 podStartE2EDuration="3.49565423s" totalImagesPullingTime="373.339609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:13.493643774 +0000 UTC m=+1573.715679316" watchObservedRunningTime="2026-10-31 00:45:13.49565423 +0000 UTC m=+1573.717689772"
Oct 31 00:45:14.593065 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:14.593033    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws"
Oct 31 00:45:14.685863 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:14.685765    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6cf9baa-9f6f-450f-be46-557469f45fde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6cf9baa-9f6f-450f-be46-557469f45fde-maas-api-openshift-ca\") pod \"a6cf9baa-9f6f-450f-be46-557469f45fde\" (UID: \"a6cf9baa-9f6f-450f-be46-557469f45fde\") "
Oct 31 00:45:14.685863 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:14.685811    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6cf9baa-9f6f-450f-be46-557469f45fde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6cf9baa-9f6f-450f-be46-557469f45fde-maas-api-xks-ca\") pod \"a6cf9baa-9f6f-450f-be46-557469f45fde\" (UID: \"a6cf9baa-9f6f-450f-be46-557469f45fde\") "
Oct 31 00:45:14.685863 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:14.685863    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6cf9baa-9f6f-450f-be46-557469f45fde-kube-api-access-sxrrq\" (UniqueName: \"kubernetes.io/projected/a6cf9baa-9f6f-450f-be46-557469f45fde-kube-api-access-sxrrq\") pod \"a6cf9baa-9f6f-450f-be46-557469f45fde\" (UID: \"a6cf9baa-9f6f-450f-be46-557469f45fde\") "
Oct 31 00:45:14.686257 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:14.686225    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6cf9baa-9f6f-450f-be46-557469f45fde-maas-api-openshift-ca" pod "a6cf9baa-9f6f-450f-be46-557469f45fde" (UID: "a6cf9baa-9f6f-450f-be46-557469f45fde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:45:14.686809 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:14.686785    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6cf9baa-9f6f-450f-be46-557469f45fde-maas-api-openshift-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:45:14.688723 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:14.688696    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6cf9baa-9f6f-450f-be46-557469f45fde-kube-api-access-sxrrq" pod "a6cf9baa-9f6f-450f-be46-557469f45fde" (UID: "a6cf9baa-9f6f-450f-be46-557469f45fde"). InnerVolumeSpecName "kube-api-access-sxrrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:45:14.688844 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:14.688715    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6cf9baa-9f6f-450f-be46-557469f45fde-maas-api-xks-ca" pod "a6cf9baa-9f6f-450f-be46-557469f45fde" (UID: "a6cf9baa-9f6f-450f-be46-557469f45fde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:45:14.788892 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:14.788851    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxrrq\" (UniqueName: \"kubernetes.io/projected/a6cf9baa-9f6f-450f-be46-557469f45fde-kube-api-access-sxrrq\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:45:14.788892 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:14.788886    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6cf9baa-9f6f-450f-be46-557469f45fde-maas-api-xks-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:45:15.294547 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:45:15.294499    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:45:15.427867 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:15.427834    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws"
Oct 31 00:45:15.427867 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:15.427855    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws" event={"ID":"a6cf9baa-9f6f-450f-be46-557469f45fde","Type":"ContainerDied","Data":"d515c5555e3a0f03905a63582411f7edbcc003fb9a0906465358a81ac75e747e"}
Oct 31 00:45:15.428116 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:15.427945    4327 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d515c5555e3a0f03905a63582411f7edbcc003fb9a0906465358a81ac75e747e"
Oct 31 00:45:18.632468 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:18.632057    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:18.825228 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:18.823796    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:45:18.887323 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:18.884409    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 00:45:18.945707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:18.913785    4327 scope.go:122] "RemoveContainer" containerID="c65ee404405e6a48ca1528fd7cf1041ca7df374854b250894944cf6b9cfb9837"
Oct 31 00:45:19.149212 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:19.149072    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:19.363702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:19.342209    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded"
Oct 31 00:45:19.496716 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:19.487845    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:19.645452 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:19.645332    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 00:45:19.647103 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:19.647071    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:19.649474 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:19.649455    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:45:19.708759 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:19.708202    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:45:19.895329 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:19.895277    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:19.909912 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:19.905921    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 00:45:19.909912 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:19.906558    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:45:19.909912 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:19.906640    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:45:20.225703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:20.201843    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:20.225703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:20.216770    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded"
Oct 31 00:45:20.710665 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:20.685768    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:20.865998 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:20.857120    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:20.947015 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:20.946960    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:20.981362 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:20.978780    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:20.990705 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:45:20.982806    4327 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.018s"
Oct 31 00:45:21.423857 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:21.423799    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:21.528705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:21.519557    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:45:21.618709 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:21.585819    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:45:21.637707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:21.619053    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:45:21.689945 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:21.689824    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 00:45:21.741722 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:21.730753    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg"
Oct 31 00:45:22.044769 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:22.025858    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:22.044769 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:22.026166    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:22.044769 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:22.026445    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:22.110101 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:21.723768    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:22.272818 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:22.253126    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded"
Oct 31 00:45:22.272818 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:22.254498    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4qhrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:45:22.272818 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:22.254543    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd" podUID="6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:22.560992 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:22.536785    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:22.560992 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:22.537195    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj" podUID="b1988c46-0b7f-4edd-b441-ab408ad0a246" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:22.590363 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:22.585139    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hk9qf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:45:22.590363 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:22.585210    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf" podUID="a5f7f289-ef35-494d-91c9-034c7f1294d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:22.794530 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:22.794485    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 00:45:23.145774 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:23.104466    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:23.145774 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:23.111758    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:23.145774 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:23.141265    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:45:23.145774 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:23.141629    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 00:45:23.228702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:23.162333    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:45:23.508767 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:23.503802    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mbzdk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:45:23.508767 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:23.503874    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mbzdk" podUID="738f0544-ef43-4c6c-a9ec-a20a362c482d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:23.580730 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:23.578899    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:23.754603 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:23.754569    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6"]
Oct 31 00:45:23.865765 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:23.830042    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:24.147118 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:24.126803    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:24.147118 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:24.128611    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:24.207704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:24.176617    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:45:24.208754 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:24.208707    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 00:45:25.079167 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:45:25.078540    4327 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.095s"
Oct 31 00:45:25.081689 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:25.081627    4327 generic.go:356] "Generic (PLEG): container finished" podID="e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f" containerID="7b4e80aa32dd74e5e98e693366327a6fe909bfaafe07ab9be391abda259b98cf" exitCode=60
Oct 31 00:45:25.123704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:25.094450    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf" event={"ID":"e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f","Type":"ContainerDied","Data":"7b4e80aa32dd74e5e98e693366327a6fe909bfaafe07ab9be391abda259b98cf"}
Oct 31 00:45:25.123704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:25.094612    4327 scope.go:122] "RemoveContainer" containerID="c65ee404405e6a48ca1528fd7cf1041ca7df374854b250894944cf6b9cfb9837"
Oct 31 00:45:25.123704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:25.095344    4327 scope.go:122] "RemoveContainer" containerID="7b4e80aa32dd74e5e98e693366327a6fe909bfaafe07ab9be391abda259b98cf"
Oct 31 00:45:25.123704 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:45:25.096018    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf_odh-ai-gateway-infra(e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf" podUID="e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f"
Oct 31 00:45:25.141035 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:25.125156    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-dddkj" podUID="b1988c46-0b7f-4edd-b441-ab408ad0a246" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 00:45:25.309702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:25.292169    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 00:45:25.341182 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:25.341056    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:25.503705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:25.342874    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:45:25.507085 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:25.507022    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt" podUID="c1398a6a-1ac9-48e1-a988-19c575e0cb91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:25.523850 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:25.399560    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8zxnz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:45:25.524174 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:25.524126    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz" podUID="820c95b2-cfd6-4893-80ed-1c746d4cfef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:25.524383 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:25.440426    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:45:25.557733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:25.343212    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 00:45:25.622124 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:25.622077    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj" podUID="cad8e534-8589-407a-bbcf-0bb145b80233" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:25.861053 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:25.861003    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 00:45:26.096500 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:26.096451    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded"
Oct 31 00:45:26.315969 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:26.315875    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:26.472871 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:26.472466    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:26.485471 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:26.485423    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:26.548091 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:26.547742    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:45:26.548801 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:26.548382    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:45:26.548801 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:26.548440    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:45:26.548801 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:26.548476    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:45:26.550140 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:26.550114    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:45:26.583603 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:26.583514    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf"]
Oct 31 00:45:26.590855 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:26.590824    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:45:27.385544 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:27.385511    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf"
Oct 31 00:45:27.581233 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:27.581141    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f-kube-api-access-d477l\" (UniqueName: \"kubernetes.io/projected/e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f-kube-api-access-d477l\") pod \"e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f\" (UID: \"e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f\") "
Oct 31 00:45:27.581423 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:27.581394    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f-maas-api-service-ca\") pod \"e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f\" (UID: \"e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f\") "
Oct 31 00:45:27.581752 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:27.581725    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f-maas-api-service-ca" pod "e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f" (UID: "e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:45:27.582537 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:27.582515    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f-maas-api-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:45:27.585767 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:27.583328    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f-kube-api-access-d477l" pod "e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f" (UID: "e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f"). InnerVolumeSpecName "kube-api-access-d477l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:45:27.616436 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:27.616415    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf"
Oct 31 00:45:27.616578 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:27.616450    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf" event={"ID":"e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f","Type":"ContainerDied","Data":"e3a1ff2362a1af4d7ac3e59c4898f7a47dd2fc0f030758aa0e67a35956ac717e"}
Oct 31 00:45:27.616578 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:27.616533    4327 scope.go:122] "RemoveContainer" containerID="7b4e80aa32dd74e5e98e693366327a6fe909bfaafe07ab9be391abda259b98cf"
Oct 31 00:45:27.683865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:27.683840    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d477l\" (UniqueName: \"kubernetes.io/projected/e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f-kube-api-access-d477l\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:45:27.776736 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:27.776642    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf"]
Oct 31 00:45:27.780627 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:27.780596    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d66e61c1-29890125-4m9mf"]
Oct 31 00:45:27.988843 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:27.988772    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f" path="/var/lib/kubelet/pods/e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f/volumes"
Oct 31 00:45:28.756702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:28.755753    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6" podUID="be6ede26-6ab9-4185-a5c2-793692a81e3d" containerName="payload-processing" containerID="cri-o://3d3bee6ecfd3fb23a8c1476392619c7b4ca87065a1a744626cad0ec91ed901df" gracePeriod=25
Oct 31 00:45:29.403658 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:29.403624    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6"
Oct 31 00:45:29.498895 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:29.498857    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be6ede26-6ab9-4185-a5c2-793692a81e3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6ede26-6ab9-4185-a5c2-793692a81e3d-plugins-config-volume\") pod \"be6ede26-6ab9-4185-a5c2-793692a81e3d\" (UID: \"be6ede26-6ab9-4185-a5c2-793692a81e3d\") "
Oct 31 00:45:29.499086 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:29.498968    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be6ede26-6ab9-4185-a5c2-793692a81e3d-kube-api-access-46fgv\" (UniqueName: \"kubernetes.io/projected/be6ede26-6ab9-4185-a5c2-793692a81e3d-kube-api-access-46fgv\") pod \"be6ede26-6ab9-4185-a5c2-793692a81e3d\" (UID: \"be6ede26-6ab9-4185-a5c2-793692a81e3d\") "
Oct 31 00:45:29.499284 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:29.499258    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be6ede26-6ab9-4185-a5c2-793692a81e3d-plugins-config-volume" pod "be6ede26-6ab9-4185-a5c2-793692a81e3d" (UID: "be6ede26-6ab9-4185-a5c2-793692a81e3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:45:29.499804 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:29.499780    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6ede26-6ab9-4185-a5c2-793692a81e3d-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:45:29.501369 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:29.501337    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6ede26-6ab9-4185-a5c2-793692a81e3d-kube-api-access-46fgv" pod "be6ede26-6ab9-4185-a5c2-793692a81e3d" (UID: "be6ede26-6ab9-4185-a5c2-793692a81e3d"). InnerVolumeSpecName "kube-api-access-46fgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:45:29.601206 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:29.601169    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46fgv\" (UniqueName: \"kubernetes.io/projected/be6ede26-6ab9-4185-a5c2-793692a81e3d-kube-api-access-46fgv\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:45:29.684926 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:29.684837    4327 generic.go:356] "Generic (PLEG): container finished" podID="be6ede26-6ab9-4185-a5c2-793692a81e3d" containerID="3d3bee6ecfd3fb23a8c1476392619c7b4ca87065a1a744626cad0ec91ed901df" exitCode=0
Oct 31 00:45:29.684926 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:29.684895    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6" event={"ID":"be6ede26-6ab9-4185-a5c2-793692a81e3d","Type":"ContainerDied","Data":"3d3bee6ecfd3fb23a8c1476392619c7b4ca87065a1a744626cad0ec91ed901df"}
Oct 31 00:45:29.684926 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:29.684910    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6"
Oct 31 00:45:29.685208 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:29.684989    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6" event={"ID":"be6ede26-6ab9-4185-a5c2-793692a81e3d","Type":"ContainerDied","Data":"5d10ebfc6be0ba7b98bc5759461611d079f8e182d4727eb7d5be66f65c53aa24"}
Oct 31 00:45:29.685208 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:29.685020    4327 scope.go:122] "RemoveContainer" containerID="3d3bee6ecfd3fb23a8c1476392619c7b4ca87065a1a744626cad0ec91ed901df"
Oct 31 00:45:29.696248 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:29.696223    4327 scope.go:122] "RemoveContainer" containerID="3d3bee6ecfd3fb23a8c1476392619c7b4ca87065a1a744626cad0ec91ed901df"
Oct 31 00:45:29.696586 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:45:29.696554    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d3bee6ecfd3fb23a8c1476392619c7b4ca87065a1a744626cad0ec91ed901df\": container with ID starting with 3d3bee6ecfd3fb23a8c1476392619c7b4ca87065a1a744626cad0ec91ed901df not found: ID does not exist" containerID="3d3bee6ecfd3fb23a8c1476392619c7b4ca87065a1a744626cad0ec91ed901df"
Oct 31 00:45:29.696673 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:29.696598    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d3bee6ecfd3fb23a8c1476392619c7b4ca87065a1a744626cad0ec91ed901df"} err="failed to get container status \"3d3bee6ecfd3fb23a8c1476392619c7b4ca87065a1a744626cad0ec91ed901df\": rpc error: code = NotFound desc = could not find container \"3d3bee6ecfd3fb23a8c1476392619c7b4ca87065a1a744626cad0ec91ed901df\": container with ID starting with 3d3bee6ecfd3fb23a8c1476392619c7b4ca87065a1a744626cad0ec91ed901df not found: ID does not exist"
Oct 31 00:45:29.748334 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:29.748295    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6"]
Oct 31 00:45:29.754786 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:29.754753    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d66e61c1-6bc4dd4f59-svst6"]
Oct 31 00:45:29.924159 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:29.924124    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be6ede26-6ab9-4185-a5c2-793692a81e3d" path="/var/lib/kubelet/pods/be6ede26-6ab9-4185-a5c2-793692a81e3d/volumes"
Oct 31 00:45:32.413071 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:32.408795    4327 patch_prober.go:29] interesting pod/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:45:32.413071 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:32.408953    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78" podUID="87c3205a-74bd-42f8-98cc-114cf902ed9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:33.653861 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:33.653815    4327 patch_prober.go:29] interesting pod/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:45:33.654335 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:33.653890    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq" podUID="71c6ffbe-9a30-4a28-aa62-57ecb4052308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:35.045348 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:35.045311    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws"]
Oct 31 00:45:35.070130 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:35.049278    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d66e61c1-sdqws"]
Oct 31 00:45:35.584121 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:35.584091    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm"]
Oct 31 00:45:35.768273 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:35.768183    4327 generic.go:356] "Generic (PLEG): container finished" podID="71c6ffbe-9a30-4a28-aa62-57ecb4052308" containerID="ed985d03890e6cbe97135b8762d6b235f915f74f8855483228b907567d7d9584" exitCode=0
Oct 31 00:45:35.768273 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:35.768253    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq" event={"ID":"71c6ffbe-9a30-4a28-aa62-57ecb4052308","Type":"ContainerDied","Data":"ed985d03890e6cbe97135b8762d6b235f915f74f8855483228b907567d7d9584"}
Oct 31 00:45:35.768808 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:35.768791    4327 scope.go:122] "RemoveContainer" containerID="ed985d03890e6cbe97135b8762d6b235f915f74f8855483228b907567d7d9584"
Oct 31 00:45:35.770427 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:35.770405    4327 generic.go:356] "Generic (PLEG): container finished" podID="87c3205a-74bd-42f8-98cc-114cf902ed9d" containerID="08522a0fe2d8b3c408453001fe6846509f241eee7c6b7beb8cec67b9ff942654" exitCode=0
Oct 31 00:45:35.770549 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:35.770469    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78" event={"ID":"87c3205a-74bd-42f8-98cc-114cf902ed9d","Type":"ContainerDied","Data":"08522a0fe2d8b3c408453001fe6846509f241eee7c6b7beb8cec67b9ff942654"}
Oct 31 00:45:35.770641 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:35.770610    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:45:35.770748 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:35.770710    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:45:35.770795 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:35.770776    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:45:35.770878 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:35.770862    4327 scope.go:122] "RemoveContainer" containerID="08522a0fe2d8b3c408453001fe6846509f241eee7c6b7beb8cec67b9ff942654"
Oct 31 00:45:35.772242 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:35.772204    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:45:35.772361 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:35.772281    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:45:35.772361 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:35.772316    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:45:36.125364 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:36.125325    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6cf9baa-9f6f-450f-be46-557469f45fde" path="/var/lib/kubelet/pods/a6cf9baa-9f6f-450f-be46-557469f45fde/volumes"
Oct 31 00:45:36.723714 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:36.717018    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"]
Oct 31 00:45:37.287701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:37.281129    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78" event={"ID":"87c3205a-74bd-42f8-98cc-114cf902ed9d","Type":"ContainerStarted","Data":"b56071a71c8ae9252f52c3c20e0f5b21760b4a8eff3a5145e01442eb3f7985d4"}
Oct 31 00:45:37.307737 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:37.305606    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq" event={"ID":"71c6ffbe-9a30-4a28-aa62-57ecb4052308","Type":"ContainerStarted","Data":"1630d6a3c905b9d7bdcf4ecc1ecfece7d4230c86ef52855bb23a60b9c993506e"}
Oct 31 00:45:37.307737 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:37.305723    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq" podUID="71c6ffbe-9a30-4a28-aa62-57ecb4052308" containerName="istio-proxy" containerID="cri-o://1630d6a3c905b9d7bdcf4ecc1ecfece7d4230c86ef52855bb23a60b9c993506e" gracePeriod=30
Oct 31 00:45:37.965909 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:37.965874    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:45:37.967327 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:37.967294    4327 patch_prober.go:29] interesting pod/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 00:45:37.967474 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:37.967357    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78" podUID="87c3205a-74bd-42f8-98cc-114cf902ed9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 00:45:38.966020 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:38.965972    4327 patch_prober.go:29] interesting pod/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 00:45:38.966422 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:38.966035    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78" podUID="87c3205a-74bd-42f8-98cc-114cf902ed9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 00:45:39.965543 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:39.965505    4327 patch_prober.go:29] interesting pod/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 00:45:39.965760 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:39.965563    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78" podUID="87c3205a-74bd-42f8-98cc-114cf902ed9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 00:45:40.585051 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:40.585011    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm" podUID="06b1f2ea-dbf5-44de-adeb-e763db1ee2ce" containerName="payload-processing" containerID="cri-o://b3214075827d05add29b445423c2187f0c9f7c26fa62fcbc40f69c0bf217a1a9" gracePeriod=25
Oct 31 00:45:40.949454 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:40.949423    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm"
Oct 31 00:45:40.964952 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:40.964919    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06b1f2ea-dbf5-44de-adeb-e763db1ee2ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b1f2ea-dbf5-44de-adeb-e763db1ee2ce-plugins-config-volume\") pod \"06b1f2ea-dbf5-44de-adeb-e763db1ee2ce\" (UID: \"06b1f2ea-dbf5-44de-adeb-e763db1ee2ce\") "
Oct 31 00:45:40.965092 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:40.965044    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06b1f2ea-dbf5-44de-adeb-e763db1ee2ce-kube-api-access-kcbwt\" (UniqueName: \"kubernetes.io/projected/06b1f2ea-dbf5-44de-adeb-e763db1ee2ce-kube-api-access-kcbwt\") pod \"06b1f2ea-dbf5-44de-adeb-e763db1ee2ce\" (UID: \"06b1f2ea-dbf5-44de-adeb-e763db1ee2ce\") "
Oct 31 00:45:40.965210 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:40.965188    4327 patch_prober.go:29] interesting pod/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 00:45:40.965261 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:40.965242    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78" podUID="87c3205a-74bd-42f8-98cc-114cf902ed9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 00:45:40.965351 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:40.965331    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06b1f2ea-dbf5-44de-adeb-e763db1ee2ce-plugins-config-volume" pod "06b1f2ea-dbf5-44de-adeb-e763db1ee2ce" (UID: "06b1f2ea-dbf5-44de-adeb-e763db1ee2ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:45:40.966032 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:40.966011    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b1f2ea-dbf5-44de-adeb-e763db1ee2ce-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:45:40.967240 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:40.967219    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b1f2ea-dbf5-44de-adeb-e763db1ee2ce-kube-api-access-kcbwt" pod "06b1f2ea-dbf5-44de-adeb-e763db1ee2ce" (UID: "06b1f2ea-dbf5-44de-adeb-e763db1ee2ce"). InnerVolumeSpecName "kube-api-access-kcbwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:45:41.068196 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:41.068029    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcbwt\" (UniqueName: \"kubernetes.io/projected/06b1f2ea-dbf5-44de-adeb-e763db1ee2ce-kube-api-access-kcbwt\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:45:41.378134 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:41.378088    4327 generic.go:356] "Generic (PLEG): container finished" podID="06b1f2ea-dbf5-44de-adeb-e763db1ee2ce" containerID="b3214075827d05add29b445423c2187f0c9f7c26fa62fcbc40f69c0bf217a1a9" exitCode=0
Oct 31 00:45:41.378330 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:41.378154    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm"
Oct 31 00:45:41.378330 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:41.378170    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm" event={"ID":"06b1f2ea-dbf5-44de-adeb-e763db1ee2ce","Type":"ContainerDied","Data":"b3214075827d05add29b445423c2187f0c9f7c26fa62fcbc40f69c0bf217a1a9"}
Oct 31 00:45:41.378330 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:41.378253    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm" event={"ID":"06b1f2ea-dbf5-44de-adeb-e763db1ee2ce","Type":"ContainerDied","Data":"491801eae2fe214f6c99c9280242350a5b954c29fb498d8ccf4e6d2c1876c736"}
Oct 31 00:45:41.378330 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:41.378275    4327 scope.go:122] "RemoveContainer" containerID="b3214075827d05add29b445423c2187f0c9f7c26fa62fcbc40f69c0bf217a1a9"
Oct 31 00:45:41.389081 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:41.389064    4327 scope.go:122] "RemoveContainer" containerID="b3214075827d05add29b445423c2187f0c9f7c26fa62fcbc40f69c0bf217a1a9"
Oct 31 00:45:41.389333 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:45:41.389307    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3214075827d05add29b445423c2187f0c9f7c26fa62fcbc40f69c0bf217a1a9\": container with ID starting with b3214075827d05add29b445423c2187f0c9f7c26fa62fcbc40f69c0bf217a1a9 not found: ID does not exist" containerID="b3214075827d05add29b445423c2187f0c9f7c26fa62fcbc40f69c0bf217a1a9"
Oct 31 00:45:41.389390 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:41.389343    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3214075827d05add29b445423c2187f0c9f7c26fa62fcbc40f69c0bf217a1a9"} err="failed to get container status \"b3214075827d05add29b445423c2187f0c9f7c26fa62fcbc40f69c0bf217a1a9\": rpc error: code = NotFound desc = could not find container \"b3214075827d05add29b445423c2187f0c9f7c26fa62fcbc40f69c0bf217a1a9\": container with ID starting with b3214075827d05add29b445423c2187f0c9f7c26fa62fcbc40f69c0bf217a1a9 not found: ID does not exist"
Oct 31 00:45:41.449640 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:41.449605    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm"]
Oct 31 00:45:41.462122 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:41.462091    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a9461dc0-68d7fb8fb7-xk6mm"]
Oct 31 00:45:41.972706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:41.971371    4327 patch_prober.go:29] interesting pod/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 00:45:41.972706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:41.971434    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78" podUID="87c3205a-74bd-42f8-98cc-114cf902ed9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 00:45:42.038585 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:42.038546    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06b1f2ea-dbf5-44de-adeb-e763db1ee2ce" path="/var/lib/kubelet/pods/06b1f2ea-dbf5-44de-adeb-e763db1ee2ce/volumes"
Oct 31 00:45:42.818895 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:42.818863    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:45:42.966498 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:42.966462    4327 patch_prober.go:29] interesting pod/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 00:45:42.966731 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:42.966512    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78" podUID="87c3205a-74bd-42f8-98cc-114cf902ed9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 00:45:43.044816 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.044657    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-workload-socket\") pod \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") "
Oct 31 00:45:43.044816 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.044714    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-envoy\") pod \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") "
Oct 31 00:45:43.044816 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.044744    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-credential-socket\") pod \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") "
Oct 31 00:45:43.047007 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.044883    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-data\") pod \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") "
Oct 31 00:45:43.047007 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.044918    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-token\" (UniqueName: \"kubernetes.io/projected/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-token\") pod \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") "
Oct 31 00:45:43.047007 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.045037    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-workload-certs\") pod \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") "
Oct 31 00:45:43.047007 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.045110    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istiod-ca-cert\") pod \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") "
Oct 31 00:45:43.047007 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.045247    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-podinfo\") pod \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") "
Oct 31 00:45:43.047007 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.045294    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71c6ffbe-9a30-4a28-aa62-57ecb4052308-kube-api-access-5gb55\" (UniqueName: \"kubernetes.io/projected/71c6ffbe-9a30-4a28-aa62-57ecb4052308-kube-api-access-5gb55\") pod \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\" (UID: \"71c6ffbe-9a30-4a28-aa62-57ecb4052308\") "
Oct 31 00:45:43.047007 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.045461    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-workload-socket" pod "71c6ffbe-9a30-4a28-aa62-57ecb4052308" (UID: "71c6ffbe-9a30-4a28-aa62-57ecb4052308"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:45:43.047007 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.045714    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istiod-ca-cert" pod "71c6ffbe-9a30-4a28-aa62-57ecb4052308" (UID: "71c6ffbe-9a30-4a28-aa62-57ecb4052308"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:45:43.047007 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.046002    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:45:43.047007 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.046020    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:45:43.047007 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.046164    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-data" pod "71c6ffbe-9a30-4a28-aa62-57ecb4052308" (UID: "71c6ffbe-9a30-4a28-aa62-57ecb4052308"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:45:43.047007 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.046215    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-credential-socket" pod "71c6ffbe-9a30-4a28-aa62-57ecb4052308" (UID: "71c6ffbe-9a30-4a28-aa62-57ecb4052308"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:45:43.047007 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.046992    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-workload-certs" pod "71c6ffbe-9a30-4a28-aa62-57ecb4052308" (UID: "71c6ffbe-9a30-4a28-aa62-57ecb4052308"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:45:43.048835 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.048797    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c6ffbe-9a30-4a28-aa62-57ecb4052308-kube-api-access-5gb55" pod "71c6ffbe-9a30-4a28-aa62-57ecb4052308" (UID: "71c6ffbe-9a30-4a28-aa62-57ecb4052308"). InnerVolumeSpecName "kube-api-access-5gb55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:45:43.049394 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.049370    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-envoy" pod "71c6ffbe-9a30-4a28-aa62-57ecb4052308" (UID: "71c6ffbe-9a30-4a28-aa62-57ecb4052308"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:45:43.049670 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.049651    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-token" pod "71c6ffbe-9a30-4a28-aa62-57ecb4052308" (UID: "71c6ffbe-9a30-4a28-aa62-57ecb4052308"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:45:43.050468 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.050439    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-podinfo" pod "71c6ffbe-9a30-4a28-aa62-57ecb4052308" (UID: "71c6ffbe-9a30-4a28-aa62-57ecb4052308"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:45:43.147877 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.147847    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:45:43.148073 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.148062    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:45:43.148135 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.148128    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:45:43.148185 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.148179    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:45:43.148236 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.148228    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gb55\" (UniqueName: \"kubernetes.io/projected/71c6ffbe-9a30-4a28-aa62-57ecb4052308-kube-api-access-5gb55\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:45:43.148291 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.148284    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:45:43.148337 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.148330    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c6ffbe-9a30-4a28-aa62-57ecb4052308-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:45:43.397254 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.397221    4327 generic.go:356] "Generic (PLEG): container finished" podID="71c6ffbe-9a30-4a28-aa62-57ecb4052308" containerID="1630d6a3c905b9d7bdcf4ecc1ecfece7d4230c86ef52855bb23a60b9c993506e" exitCode=0
Oct 31 00:45:43.397438 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.397419    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq" event={"ID":"71c6ffbe-9a30-4a28-aa62-57ecb4052308","Type":"ContainerDied","Data":"1630d6a3c905b9d7bdcf4ecc1ecfece7d4230c86ef52855bb23a60b9c993506e"}
Oct 31 00:45:43.397529 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.397477    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq" event={"ID":"71c6ffbe-9a30-4a28-aa62-57ecb4052308","Type":"ContainerDied","Data":"62af32dd6bfdc665c2b59530b8d89d3bca01ad2436fa866cd9729a28063ccdf9"}
Oct 31 00:45:43.397529 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.397508    4327 scope.go:122] "RemoveContainer" containerID="1630d6a3c905b9d7bdcf4ecc1ecfece7d4230c86ef52855bb23a60b9c993506e"
Oct 31 00:45:43.397657 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.397642    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"
Oct 31 00:45:43.414837 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.414814    4327 scope.go:122] "RemoveContainer" containerID="ed985d03890e6cbe97135b8762d6b235f915f74f8855483228b907567d7d9584"
Oct 31 00:45:43.433789 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.433764    4327 scope.go:122] "RemoveContainer" containerID="1630d6a3c905b9d7bdcf4ecc1ecfece7d4230c86ef52855bb23a60b9c993506e"
Oct 31 00:45:43.434147 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:45:43.434121    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1630d6a3c905b9d7bdcf4ecc1ecfece7d4230c86ef52855bb23a60b9c993506e\": container with ID starting with 1630d6a3c905b9d7bdcf4ecc1ecfece7d4230c86ef52855bb23a60b9c993506e not found: ID does not exist" containerID="1630d6a3c905b9d7bdcf4ecc1ecfece7d4230c86ef52855bb23a60b9c993506e"
Oct 31 00:45:43.434225 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.434178    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1630d6a3c905b9d7bdcf4ecc1ecfece7d4230c86ef52855bb23a60b9c993506e"} err="failed to get container status \"1630d6a3c905b9d7bdcf4ecc1ecfece7d4230c86ef52855bb23a60b9c993506e\": rpc error: code = NotFound desc = could not find container \"1630d6a3c905b9d7bdcf4ecc1ecfece7d4230c86ef52855bb23a60b9c993506e\": container with ID starting with 1630d6a3c905b9d7bdcf4ecc1ecfece7d4230c86ef52855bb23a60b9c993506e not found: ID does not exist"
Oct 31 00:45:43.434225 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.434202    4327 scope.go:122] "RemoveContainer" containerID="ed985d03890e6cbe97135b8762d6b235f915f74f8855483228b907567d7d9584"
Oct 31 00:45:43.434485 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:45:43.434466    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed985d03890e6cbe97135b8762d6b235f915f74f8855483228b907567d7d9584\": container with ID starting with ed985d03890e6cbe97135b8762d6b235f915f74f8855483228b907567d7d9584 not found: ID does not exist" containerID="ed985d03890e6cbe97135b8762d6b235f915f74f8855483228b907567d7d9584"
Oct 31 00:45:43.434538 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.434490    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed985d03890e6cbe97135b8762d6b235f915f74f8855483228b907567d7d9584"} err="failed to get container status \"ed985d03890e6cbe97135b8762d6b235f915f74f8855483228b907567d7d9584\": rpc error: code = NotFound desc = could not find container \"ed985d03890e6cbe97135b8762d6b235f915f74f8855483228b907567d7d9584\": container with ID starting with ed985d03890e6cbe97135b8762d6b235f915f74f8855483228b907567d7d9584 not found: ID does not exist"
Oct 31 00:45:43.462804 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.462766    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"]
Oct 31 00:45:43.472751 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.472716    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d66e61c1-openshift-default-77c7cff769-c69sq"]
Oct 31 00:45:43.727403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.727321    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz"]
Oct 31 00:45:43.728112 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.728089    4327 cpu_manager.go:414] "containerMap: removing container" podUID="71c6ffbe-9a30-4a28-aa62-57ecb4052308" containerName="istio-proxy"
Oct 31 00:45:43.728112 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.728112    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71c6ffbe-9a30-4a28-aa62-57ecb4052308" containerName="istio-proxy"
Oct 31 00:45:43.728306 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.728127    4327 cpu_manager.go:414] "containerMap: removing container" podUID="e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f" containerName="cleanup"
Oct 31 00:45:43.728306 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.728135    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f" containerName="cleanup"
Oct 31 00:45:43.728306 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.728163    4327 cpu_manager.go:414] "containerMap: removing container" podUID="06b1f2ea-dbf5-44de-adeb-e763db1ee2ce" containerName="payload-processing"
Oct 31 00:45:43.728306 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.728171    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06b1f2ea-dbf5-44de-adeb-e763db1ee2ce" containerName="payload-processing"
Oct 31 00:45:43.728306 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.728205    4327 cpu_manager.go:414] "containerMap: removing container" podUID="71c6ffbe-9a30-4a28-aa62-57ecb4052308" containerName="istio-proxy"
Oct 31 00:45:43.728306 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.728214    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71c6ffbe-9a30-4a28-aa62-57ecb4052308" containerName="istio-proxy"
Oct 31 00:45:43.728306 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.728233    4327 cpu_manager.go:414] "containerMap: removing container" podUID="be6ede26-6ab9-4185-a5c2-793692a81e3d" containerName="payload-processing"
Oct 31 00:45:43.728306 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.728239    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be6ede26-6ab9-4185-a5c2-793692a81e3d" containerName="payload-processing"
Oct 31 00:45:43.728306 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.728275    4327 cpu_manager.go:414] "containerMap: removing container" podUID="a6cf9baa-9f6f-450f-be46-557469f45fde" containerName="revoke-keys"
Oct 31 00:45:43.728306 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.728282    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6cf9baa-9f6f-450f-be46-557469f45fde" containerName="revoke-keys"
Oct 31 00:45:43.728306 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.728308    4327 cpu_manager.go:414] "containerMap: removing container" podUID="e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f" containerName="cleanup"
Oct 31 00:45:43.728849 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.728314    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f" containerName="cleanup"
Oct 31 00:45:43.728849 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.728474    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="71c6ffbe-9a30-4a28-aa62-57ecb4052308" containerName="istio-proxy"
Oct 31 00:45:43.728849 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.728489    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="be6ede26-6ab9-4185-a5c2-793692a81e3d" containerName="payload-processing"
Oct 31 00:45:43.728849 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.728501    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6cf9baa-9f6f-450f-be46-557469f45fde" containerName="revoke-keys"
Oct 31 00:45:43.728849 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.728512    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f" containerName="cleanup"
Oct 31 00:45:43.728849 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.728521    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f" containerName="cleanup"
Oct 31 00:45:43.728849 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.728531    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="71c6ffbe-9a30-4a28-aa62-57ecb4052308" containerName="istio-proxy"
Oct 31 00:45:43.728849 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.728541    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f" containerName="cleanup"
Oct 31 00:45:43.728849 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.728553    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="06b1f2ea-dbf5-44de-adeb-e763db1ee2ce" containerName="payload-processing"
Oct 31 00:45:43.731139 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.731118    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz"
Oct 31 00:45:43.755786 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.755755    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz"]
Oct 31 00:45:43.756386 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.756364    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-c2cd8283-dockercfg-g74sp\""
Oct 31 00:45:43.757282 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.757256    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-c2cd8283\""
Oct 31 00:45:43.766568 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.766540    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0c856c-f6b5-4d3c-ae26-84d887ee092e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz\" (UID: \"1c0c856c-f6b5-4d3c-ae26-84d887ee092e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz"
Oct 31 00:45:43.766761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.766652    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6szv\" (UniqueName: \"kubernetes.io/projected/1c0c856c-f6b5-4d3c-ae26-84d887ee092e-kube-api-access-z6szv\") pod \"payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz\" (UID: \"1c0c856c-f6b5-4d3c-ae26-84d887ee092e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz"
Oct 31 00:45:43.963738 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.952966    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0c856c-f6b5-4d3c-ae26-84d887ee092e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz\" (UID: \"1c0c856c-f6b5-4d3c-ae26-84d887ee092e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz"
Oct 31 00:45:43.963738 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:43.953231    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6szv\" (UniqueName: \"kubernetes.io/projected/1c0c856c-f6b5-4d3c-ae26-84d887ee092e-kube-api-access-z6szv\") pod \"payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz\" (UID: \"1c0c856c-f6b5-4d3c-ae26-84d887ee092e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz"
Oct 31 00:45:44.021840 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:44.021767    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0c856c-f6b5-4d3c-ae26-84d887ee092e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz\" (UID: \"1c0c856c-f6b5-4d3c-ae26-84d887ee092e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz"
Oct 31 00:45:44.140705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:44.140624    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6szv\" (UniqueName: \"kubernetes.io/projected/1c0c856c-f6b5-4d3c-ae26-84d887ee092e-kube-api-access-z6szv\") pod \"payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz\" (UID: \"1c0c856c-f6b5-4d3c-ae26-84d887ee092e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz"
Oct 31 00:45:44.310475 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:44.310380    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71c6ffbe-9a30-4a28-aa62-57ecb4052308" path="/var/lib/kubelet/pods/71c6ffbe-9a30-4a28-aa62-57ecb4052308/volumes"
Oct 31 00:45:44.311478 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:44.311020    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:45:44.311478 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:44.311076    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c2cd8283-587685db69-76crd"]
Oct 31 00:45:44.313344 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:44.313319    4327 cpu_manager.go:414] "containerMap: removing container" podUID="e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f" containerName="cleanup"
Oct 31 00:45:44.313503 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:44.313346    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e17ec3d6-2c2c-421b-a5c0-93692ca0ae4f" containerName="cleanup"
Oct 31 00:45:44.319396 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:44.319375    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c2cd8283-587685db69-76crd"]
Oct 31 00:45:44.319554 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:44.319544    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:45:44.319644 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:44.319487    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c2cd8283-587685db69-76crd"
Oct 31 00:45:44.319806 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:44.319797    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:45:44.345308 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:44.345285    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz"
Oct 31 00:45:44.423588 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:44.423545    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ndcn\" (UniqueName: \"kubernetes.io/projected/cd485e06-0892-45a9-9419-3f410e9f866d-kube-api-access-2ndcn\") pod \"payload-processing-e2e-derive-c2cd8283-587685db69-76crd\" (UID: \"cd485e06-0892-45a9-9419-3f410e9f866d\") " pod="openshift-ingress/payload-processing-e2e-derive-c2cd8283-587685db69-76crd"
Oct 31 00:45:44.423740 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:44.423654    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd485e06-0892-45a9-9419-3f410e9f866d-plugins-config-volume\") pod \"payload-processing-e2e-derive-c2cd8283-587685db69-76crd\" (UID: \"cd485e06-0892-45a9-9419-3f410e9f866d\") " pod="openshift-ingress/payload-processing-e2e-derive-c2cd8283-587685db69-76crd"
Oct 31 00:45:44.534925 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:44.532023    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ndcn\" (UniqueName: \"kubernetes.io/projected/cd485e06-0892-45a9-9419-3f410e9f866d-kube-api-access-2ndcn\") pod \"payload-processing-e2e-derive-c2cd8283-587685db69-76crd\" (UID: \"cd485e06-0892-45a9-9419-3f410e9f866d\") " pod="openshift-ingress/payload-processing-e2e-derive-c2cd8283-587685db69-76crd"
Oct 31 00:45:44.537899 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:44.537856    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd485e06-0892-45a9-9419-3f410e9f866d-plugins-config-volume\") pod \"payload-processing-e2e-derive-c2cd8283-587685db69-76crd\" (UID: \"cd485e06-0892-45a9-9419-3f410e9f866d\") " pod="openshift-ingress/payload-processing-e2e-derive-c2cd8283-587685db69-76crd"
Oct 31 00:45:44.539217 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:44.539189    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd485e06-0892-45a9-9419-3f410e9f866d-plugins-config-volume\") pod \"payload-processing-e2e-derive-c2cd8283-587685db69-76crd\" (UID: \"cd485e06-0892-45a9-9419-3f410e9f866d\") " pod="openshift-ingress/payload-processing-e2e-derive-c2cd8283-587685db69-76crd"
Oct 31 00:45:44.560559 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:44.560477    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ndcn\" (UniqueName: \"kubernetes.io/projected/cd485e06-0892-45a9-9419-3f410e9f866d-kube-api-access-2ndcn\") pod \"payload-processing-e2e-derive-c2cd8283-587685db69-76crd\" (UID: \"cd485e06-0892-45a9-9419-3f410e9f866d\") " pod="openshift-ingress/payload-processing-e2e-derive-c2cd8283-587685db69-76crd"
Oct 31 00:45:44.637269 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:44.637234    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c2cd8283-587685db69-76crd"
Oct 31 00:45:44.660284 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:45:44.660251    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0c856c_f6b5_4d3c_ae26_84d887ee092e.slice/crio-3641f93f84482551492bd83b3c1172facd096a9c4c18f49be27319daa5ef2055 WatchSource:0}: Error finding container 3641f93f84482551492bd83b3c1172facd096a9c4c18f49be27319daa5ef2055: Status 404 returned error can't find the container with id 3641f93f84482551492bd83b3c1172facd096a9c4c18f49be27319daa5ef2055
Oct 31 00:45:44.661193 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:44.661161    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz"]
Oct 31 00:45:44.978767 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:44.976619    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c2cd8283-587685db69-76crd"]
Oct 31 00:45:45.294282 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:45:45.294197    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:45:45.420298 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:45.420259    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c2cd8283-587685db69-76crd" event={"ID":"cd485e06-0892-45a9-9419-3f410e9f866d","Type":"ContainerStarted","Data":"14933828e84238d521da847a7ef6dfe942e112ec825f79fd6d1c76586bf01cf0"}
Oct 31 00:45:45.420513 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:45.420325    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c2cd8283-587685db69-76crd" event={"ID":"cd485e06-0892-45a9-9419-3f410e9f866d","Type":"ContainerStarted","Data":"25f9516bf0c5d6f6dd6b6c2c34c4bd81a55d22504999e878836bc036ee88cdc8"}
Oct 31 00:45:45.420513 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:45.420359    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-c2cd8283-587685db69-76crd"
Oct 31 00:45:45.422004 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:45.421982    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz" event={"ID":"1c0c856c-f6b5-4d3c-ae26-84d887ee092e","Type":"ContainerStarted","Data":"a9efaebde7873a537cc942468879591f69addb55dfae75cd2f8ad45beb7380a8"}
Oct 31 00:45:45.422108 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:45.422018    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz" event={"ID":"1c0c856c-f6b5-4d3c-ae26-84d887ee092e","Type":"ContainerStarted","Data":"3641f93f84482551492bd83b3c1172facd096a9c4c18f49be27319daa5ef2055"}
Oct 31 00:45:45.422187 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:45.422170    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz"
Oct 31 00:45:45.465321 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:45.465269    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-c2cd8283-587685db69-76crd" podStartSLOduration=2.465255628 podStartE2EDuration="2.465255628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:45.460933046 +0000 UTC m=+1605.682968588" watchObservedRunningTime="2026-10-31 00:45:45.465255628 +0000 UTC m=+1605.687291171"
Oct 31 00:45:45.502660 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:45.502608    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz" podStartSLOduration=2.502593466 podStartE2EDuration="2.502593466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:45.501128851 +0000 UTC m=+1605.723164567" watchObservedRunningTime="2026-10-31 00:45:45.502593466 +0000 UTC m=+1605.724629009"
Oct 31 00:45:52.201913 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:52.201873    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"]
Oct 31 00:45:52.202786 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:52.202719    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78" podUID="87c3205a-74bd-42f8-98cc-114cf902ed9d" containerName="istio-proxy" containerID="cri-o://b56071a71c8ae9252f52c3c20e0f5b21760b4a8eff3a5145e01442eb3f7985d4" gracePeriod=30
Oct 31 00:45:54.036461 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.036422    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"]
Oct 31 00:45:54.041376 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.041348    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.085042 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.085008    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-36af31-openshift-default-dockercfg-vv85m\""
Oct 31 00:45:54.093741 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.092081    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm7km\" (UniqueName: \"kubernetes.io/projected/004c3b2b-bdd7-42df-a961-3ce60631a8b4-kube-api-access-zm7km\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.093741 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.092126    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-data\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.093741 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.092222    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-envoy\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.093741 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.092274    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-workload-certs\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.093741 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.092300    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-token\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.093741 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.092343    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-podinfo\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.093741 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.092466    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-workload-socket\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.093741 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.092508    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-credential-socket\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.093741 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.092557    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istiod-ca-cert\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.098848 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.098820    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"]
Oct 31 00:45:54.193833 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.193795    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm7km\" (UniqueName: \"kubernetes.io/projected/004c3b2b-bdd7-42df-a961-3ce60631a8b4-kube-api-access-zm7km\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.193833 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.193844    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-data\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.194178 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.193945    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-envoy\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.194178 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.194000    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-workload-certs\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.194178 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.194028    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-token\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.194178 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.194073    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-podinfo\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.194378 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.194206    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-workload-socket\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.194378 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.194249    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-credential-socket\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.194378 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.194301    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istiod-ca-cert\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.194378 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.194314    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-data\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.194584 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.194388    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-workload-certs\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.194696 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.194661    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-workload-socket\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.194875 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.194850    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-credential-socket\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.195026 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.195007    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istiod-ca-cert\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.196318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.196302    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-envoy\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.196764 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.196742    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-podinfo\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.218161 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.218136    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm7km\" (UniqueName: \"kubernetes.io/projected/004c3b2b-bdd7-42df-a961-3ce60631a8b4-kube-api-access-zm7km\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.220456 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.220426    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-token\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:54.423701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:54.419011    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:55.015098 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:45:55.015069    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004c3b2b_bdd7_42df_a961_3ce60631a8b4.slice/crio-e1336d0e589fabd0eb8bb4a1aa3a03fcd0ff21e85282d1ff4806c1407775898e WatchSource:0}: Error finding container e1336d0e589fabd0eb8bb4a1aa3a03fcd0ff21e85282d1ff4806c1407775898e: Status 404 returned error can't find the container with id e1336d0e589fabd0eb8bb4a1aa3a03fcd0ff21e85282d1ff4806c1407775898e
Oct 31 00:45:55.019039 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:55.019003    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:45:55.019132 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:55.019078    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:45:55.019132 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:55.019108    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:45:55.032395 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:55.032361    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"]
Oct 31 00:45:55.760458 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:55.749004    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp" event={"ID":"004c3b2b-bdd7-42df-a961-3ce60631a8b4","Type":"ContainerStarted","Data":"c1c99c775dd03e98676e576977294ef0847c4e7acccbb1df88db67cf093a330e"}
Oct 31 00:45:55.760458 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:55.749078    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp" event={"ID":"004c3b2b-bdd7-42df-a961-3ce60631a8b4","Type":"ContainerStarted","Data":"e1336d0e589fabd0eb8bb4a1aa3a03fcd0ff21e85282d1ff4806c1407775898e"}
Oct 31 00:45:55.822762 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:55.819402    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp" podStartSLOduration=1.819379589 podStartE2EDuration="1.819379589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:55.809354392 +0000 UTC m=+1616.031389935" watchObservedRunningTime="2026-10-31 00:45:55.819379589 +0000 UTC m=+1616.041415132"
Oct 31 00:45:56.422597 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:56.422562    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:56.444494 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:56.444458    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:56.476772 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:56.475046    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz"
Oct 31 00:45:56.477230 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:56.477163    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-c2cd8283-587685db69-76crd"
Oct 31 00:45:56.985520 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:56.985486    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:56.987535 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:56.987248    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:45:58.344722 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:58.344606    4327 generic.go:356] "Generic (PLEG): container finished" podID="87c3205a-74bd-42f8-98cc-114cf902ed9d" containerID="b56071a71c8ae9252f52c3c20e0f5b21760b4a8eff3a5145e01442eb3f7985d4" exitCode=0
Oct 31 00:45:58.479823 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:58.479051    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 00:45:58.635042 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:58.634995    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:45:58.783742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:58.638184    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78" event={"ID":"87c3205a-74bd-42f8-98cc-114cf902ed9d","Type":"ContainerDied","Data":"b56071a71c8ae9252f52c3c20e0f5b21760b4a8eff3a5145e01442eb3f7985d4"}
Oct 31 00:45:58.783742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:58.749010    4327 scope.go:122] "RemoveContainer" containerID="08522a0fe2d8b3c408453001fe6846509f241eee7c6b7beb8cec67b9ff942654"
Oct 31 00:45:59.003961 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:59.003852    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:45:59.014077 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:59.014024    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:59.106718 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:59.069177    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:59.106718 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:59.070833    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 00:45:59.492872 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:59.492820    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:59.536383 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:59.498920    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:59.625718 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:59.624083    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:45:59.633715 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:59.547006    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:45:59.748443 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:59.748345    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:45:59.777704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:59.750227    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:45:59.906506 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:59.839876    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-token\" (UniqueName: \"kubernetes.io/projected/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-token\") pod \"87c3205a-74bd-42f8-98cc-114cf902ed9d\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") "
Oct 31 00:45:59.906897 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:59.906874    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87c3205a-74bd-42f8-98cc-114cf902ed9d-kube-api-access-4td7l\" (UniqueName: \"kubernetes.io/projected/87c3205a-74bd-42f8-98cc-114cf902ed9d-kube-api-access-4td7l\") pod \"87c3205a-74bd-42f8-98cc-114cf902ed9d\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") "
Oct 31 00:45:59.907127 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:59.907109    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-data\") pod \"87c3205a-74bd-42f8-98cc-114cf902ed9d\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") "
Oct 31 00:45:59.907323 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:59.907307    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-envoy\") pod \"87c3205a-74bd-42f8-98cc-114cf902ed9d\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") "
Oct 31 00:45:59.907473 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:59.907459    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-workload-socket\") pod \"87c3205a-74bd-42f8-98cc-114cf902ed9d\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") "
Oct 31 00:45:59.907639 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:59.907624    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-credential-socket\") pod \"87c3205a-74bd-42f8-98cc-114cf902ed9d\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") "
Oct 31 00:45:59.907933 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:59.907914    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87c3205a-74bd-42f8-98cc-114cf902ed9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c3205a-74bd-42f8-98cc-114cf902ed9d-istiod-ca-cert\") pod \"87c3205a-74bd-42f8-98cc-114cf902ed9d\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") "
Oct 31 00:45:59.922137 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:45:59.908332    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-podinfo\") pod \"87c3205a-74bd-42f8-98cc-114cf902ed9d\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") "
Oct 31 00:46:00.004742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:00.003027    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-data" pod "87c3205a-74bd-42f8-98cc-114cf902ed9d" (UID: "87c3205a-74bd-42f8-98cc-114cf902ed9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:46:00.033706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:00.020216    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:46:00.033706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:00.020713    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:46:00.108719 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:00.088036    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:46:00.132973 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:00.090775    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:46:00.133204 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:00.091716    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-workload-certs\") pod \"87c3205a-74bd-42f8-98cc-114cf902ed9d\" (UID: \"87c3205a-74bd-42f8-98cc-114cf902ed9d\") "
Oct 31 00:46:00.133711 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:00.133657    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-workload-certs" pod "87c3205a-74bd-42f8-98cc-114cf902ed9d" (UID: "87c3205a-74bd-42f8-98cc-114cf902ed9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:46:00.297734 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:00.297631    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-workload-socket" pod "87c3205a-74bd-42f8-98cc-114cf902ed9d" (UID: "87c3205a-74bd-42f8-98cc-114cf902ed9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:46:00.298165 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:00.298141    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-credential-socket" pod "87c3205a-74bd-42f8-98cc-114cf902ed9d" (UID: "87c3205a-74bd-42f8-98cc-114cf902ed9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:46:00.298596 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:00.298578    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87c3205a-74bd-42f8-98cc-114cf902ed9d-istiod-ca-cert" pod "87c3205a-74bd-42f8-98cc-114cf902ed9d" (UID: "87c3205a-74bd-42f8-98cc-114cf902ed9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:46:00.464111 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:00.463917    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-podinfo" pod "87c3205a-74bd-42f8-98cc-114cf902ed9d" (UID: "87c3205a-74bd-42f8-98cc-114cf902ed9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:46:00.646432 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:00.640882    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 00:46:00.681746 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:00.681706    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-token" pod "87c3205a-74bd-42f8-98cc-114cf902ed9d" (UID: "87c3205a-74bd-42f8-98cc-114cf902ed9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:46:00.799023 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:00.766836    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:46:01.032645 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.032602    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:46:01.082775 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.045578    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt" podUID="c1398a6a-1ac9-48e1-a988-19c575e0cb91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:46:01.083715 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.046912    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"
Oct 31 00:46:01.098164 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.098131    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:46:01.148152 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.148120    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:46:01.149789 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.149762    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c3205a-74bd-42f8-98cc-114cf902ed9d-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:46:01.149959 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.149943    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:46:01.150083 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.150068    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:46:01.150172 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.150161    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:46:01.150276 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.150263    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:46:01.150363 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.148651    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c3205a-74bd-42f8-98cc-114cf902ed9d-kube-api-access-4td7l" pod "87c3205a-74bd-42f8-98cc-114cf902ed9d" (UID: "87c3205a-74bd-42f8-98cc-114cf902ed9d"). InnerVolumeSpecName "kube-api-access-4td7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:46:01.168968 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.168931    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-envoy" pod "87c3205a-74bd-42f8-98cc-114cf902ed9d" (UID: "87c3205a-74bd-42f8-98cc-114cf902ed9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:46:01.296760 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.273245    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4td7l\" (UniqueName: \"kubernetes.io/projected/87c3205a-74bd-42f8-98cc-114cf902ed9d-kube-api-access-4td7l\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:46:01.296760 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.273277    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c3205a-74bd-42f8-98cc-114cf902ed9d-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:46:01.296760 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:46:01.274859    4327 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.354s"
Oct 31 00:46:01.296760 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.274928    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:46:01.296760 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.275044    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:46:01.296760 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.275070    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78" event={"ID":"87c3205a-74bd-42f8-98cc-114cf902ed9d","Type":"ContainerDied","Data":"c7d034c582cac510996f151999be31b82a1455b4b1bd0a208a57ea9e70fab092"}
Oct 31 00:46:01.296760 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.275130    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:46:01.296760 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.275170    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:46:01.296760 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.275199    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:46:01.296760 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.275231    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:46:01.296760 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.275961    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:46:01.296760 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.276264    4327 scope.go:122] "RemoveContainer" containerID="b56071a71c8ae9252f52c3c20e0f5b21760b4a8eff3a5145e01442eb3f7985d4"
Oct 31 00:46:01.374036 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.373528    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 00:46:01.376972 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.376869    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 00:46:01.378432 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.378386    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 00:46:01.885627 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.885594    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"]
Oct 31 00:46:01.895315 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.891606    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-f598cd05-gw-openshift-default-69c7fc7665-6sd78"]
Oct 31 00:46:01.933708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.933050    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87c3205a-74bd-42f8-98cc-114cf902ed9d" path="/var/lib/kubelet/pods/87c3205a-74bd-42f8-98cc-114cf902ed9d/volumes"
Oct 31 00:46:01.953057 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.953021    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b"]
Oct 31 00:46:01.954133 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.954097    4327 cpu_manager.go:414] "containerMap: removing container" podUID="87c3205a-74bd-42f8-98cc-114cf902ed9d" containerName="istio-proxy"
Oct 31 00:46:01.954133 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.954120    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87c3205a-74bd-42f8-98cc-114cf902ed9d" containerName="istio-proxy"
Oct 31 00:46:01.954297 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.954177    4327 cpu_manager.go:414] "containerMap: removing container" podUID="87c3205a-74bd-42f8-98cc-114cf902ed9d" containerName="istio-proxy"
Oct 31 00:46:01.954297 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.954185    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87c3205a-74bd-42f8-98cc-114cf902ed9d" containerName="istio-proxy"
Oct 31 00:46:01.954360 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.954299    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="87c3205a-74bd-42f8-98cc-114cf902ed9d" containerName="istio-proxy"
Oct 31 00:46:01.957785 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.957761    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b"
Oct 31 00:46:01.977080 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.976749    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-36af31\""
Oct 31 00:46:01.977080 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.977033    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-36af31-dockercfg-m9hj9\""
Oct 31 00:46:01.981645 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:01.981622    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b"]
Oct 31 00:46:02.546703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:02.521552    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/906ce4bc-762e-461b-99f1-3109cc41c2ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b\" (UID: \"906ce4bc-762e-461b-99f1-3109cc41c2ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b"
Oct 31 00:46:02.641734 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:02.579368    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlfg8\" (UniqueName: \"kubernetes.io/projected/906ce4bc-762e-461b-99f1-3109cc41c2ff-kube-api-access-tlfg8\") pod \"payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b\" (UID: \"906ce4bc-762e-461b-99f1-3109cc41c2ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b"
Oct 31 00:46:02.692660 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:02.692292    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb"]
Oct 31 00:46:02.710192 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:02.707097    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="87c3205a-74bd-42f8-98cc-114cf902ed9d" containerName="istio-proxy"
Oct 31 00:46:02.711761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:02.711732    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb"]
Oct 31 00:46:02.711901 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:02.711845    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb"
Oct 31 00:46:02.806804 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:02.806658    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rtbk\" (UniqueName: \"kubernetes.io/projected/c567b551-e6b1-421a-af55-89097b32d4ff-kube-api-access-8rtbk\") pod \"payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb\" (UID: \"c567b551-e6b1-421a-af55-89097b32d4ff\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb"
Oct 31 00:46:02.806993 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:02.806896    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c567b551-e6b1-421a-af55-89097b32d4ff-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb\" (UID: \"c567b551-e6b1-421a-af55-89097b32d4ff\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb"
Oct 31 00:46:02.807129 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:02.807112    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/906ce4bc-762e-461b-99f1-3109cc41c2ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b\" (UID: \"906ce4bc-762e-461b-99f1-3109cc41c2ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b"
Oct 31 00:46:02.807182 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:02.807137    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlfg8\" (UniqueName: \"kubernetes.io/projected/906ce4bc-762e-461b-99f1-3109cc41c2ff-kube-api-access-tlfg8\") pod \"payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b\" (UID: \"906ce4bc-762e-461b-99f1-3109cc41c2ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b"
Oct 31 00:46:02.807999 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:02.807978    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/906ce4bc-762e-461b-99f1-3109cc41c2ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b\" (UID: \"906ce4bc-762e-461b-99f1-3109cc41c2ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b"
Oct 31 00:46:02.843091 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:02.842973    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlfg8\" (UniqueName: \"kubernetes.io/projected/906ce4bc-762e-461b-99f1-3109cc41c2ff-kube-api-access-tlfg8\") pod \"payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b\" (UID: \"906ce4bc-762e-461b-99f1-3109cc41c2ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b"
Oct 31 00:46:02.871066 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:02.871034    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b"
Oct 31 00:46:02.909325 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:02.909276    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rtbk\" (UniqueName: \"kubernetes.io/projected/c567b551-e6b1-421a-af55-89097b32d4ff-kube-api-access-8rtbk\") pod \"payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb\" (UID: \"c567b551-e6b1-421a-af55-89097b32d4ff\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb"
Oct 31 00:46:02.909754 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:02.909400    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c567b551-e6b1-421a-af55-89097b32d4ff-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb\" (UID: \"c567b551-e6b1-421a-af55-89097b32d4ff\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb"
Oct 31 00:46:02.910211 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:02.910187    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c567b551-e6b1-421a-af55-89097b32d4ff-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb\" (UID: \"c567b551-e6b1-421a-af55-89097b32d4ff\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb"
Oct 31 00:46:02.947382 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:02.947335    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rtbk\" (UniqueName: \"kubernetes.io/projected/c567b551-e6b1-421a-af55-89097b32d4ff-kube-api-access-8rtbk\") pod \"payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb\" (UID: \"c567b551-e6b1-421a-af55-89097b32d4ff\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb"
Oct 31 00:46:03.032673 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:03.031799    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb"
Oct 31 00:46:03.249168 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:03.249118    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b"]
Oct 31 00:46:03.255119 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:46:03.255085    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod906ce4bc_762e_461b_99f1_3109cc41c2ff.slice/crio-5b5ad490b0bd35d3dbbd612f33a7f34523b8158d8861738f639dde0863a47f7f WatchSource:0}: Error finding container 5b5ad490b0bd35d3dbbd612f33a7f34523b8158d8861738f639dde0863a47f7f: Status 404 returned error can't find the container with id 5b5ad490b0bd35d3dbbd612f33a7f34523b8158d8861738f639dde0863a47f7f
Oct 31 00:46:03.366535 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:03.366500    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb"]
Oct 31 00:46:03.367523 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:46:03.367495    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc567b551_e6b1_421a_af55_89097b32d4ff.slice/crio-19cfc85da4270797c37cfeb6f306d72ebbcf06cd956d0bc46101f626f2524982 WatchSource:0}: Error finding container 19cfc85da4270797c37cfeb6f306d72ebbcf06cd956d0bc46101f626f2524982: Status 404 returned error can't find the container with id 19cfc85da4270797c37cfeb6f306d72ebbcf06cd956d0bc46101f626f2524982
Oct 31 00:46:03.894706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:03.894512    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b" event={"ID":"906ce4bc-762e-461b-99f1-3109cc41c2ff","Type":"ContainerStarted","Data":"b9e88fc672b332fc0b8a2acb94b75dc7a481d85c72db7426438828777e1d7bed"}
Oct 31 00:46:03.894706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:03.894583    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b" event={"ID":"906ce4bc-762e-461b-99f1-3109cc41c2ff","Type":"ContainerStarted","Data":"5b5ad490b0bd35d3dbbd612f33a7f34523b8158d8861738f639dde0863a47f7f"}
Oct 31 00:46:03.895651 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:03.895584    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b"
Oct 31 00:46:03.911548 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:03.910378    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb" event={"ID":"c567b551-e6b1-421a-af55-89097b32d4ff","Type":"ContainerStarted","Data":"d7ebe83d27f827b15ea28cca513c6c19f4447dde7056648355d9c49262b71335"}
Oct 31 00:46:03.911548 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:03.910452    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb" event={"ID":"c567b551-e6b1-421a-af55-89097b32d4ff","Type":"ContainerStarted","Data":"19cfc85da4270797c37cfeb6f306d72ebbcf06cd956d0bc46101f626f2524982"}
Oct 31 00:46:03.911548 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:03.911491    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb"
Oct 31 00:46:03.977508 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:03.977075    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b" podStartSLOduration=2.977056438 podStartE2EDuration="2.977056438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:03.976219516 +0000 UTC m=+1624.198255059" watchObservedRunningTime="2026-10-31 00:46:03.977056438 +0000 UTC m=+1624.199091980"
Oct 31 00:46:04.019775 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:04.019714    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb" podStartSLOduration=2.019694708 podStartE2EDuration="2.019694708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:04.018019988 +0000 UTC m=+1624.240055531" watchObservedRunningTime="2026-10-31 00:46:04.019694708 +0000 UTC m=+1624.241730290"
Oct 31 00:46:05.656748 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:05.656705    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg"]
Oct 31 00:46:05.658531 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:05.658497    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg" podUID="191c12a0-fb78-4586-8d87-80f3cf3dce22" containerName="maas-api" containerID="cri-o://3311a8a035a6f8553a080ac0db77b89b5e167b96ac82c9f022ac9e400979003a" gracePeriod=30
Oct 31 00:46:06.183821 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:06.183458    4327 generic.go:356] "Generic (PLEG): container finished" podID="191c12a0-fb78-4586-8d87-80f3cf3dce22" containerID="3311a8a035a6f8553a080ac0db77b89b5e167b96ac82c9f022ac9e400979003a" exitCode=0
Oct 31 00:46:06.191184 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:06.184646    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg" event={"ID":"191c12a0-fb78-4586-8d87-80f3cf3dce22","Type":"ContainerDied","Data":"3311a8a035a6f8553a080ac0db77b89b5e167b96ac82c9f022ac9e400979003a"}
Oct 31 00:46:06.191184 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:06.187039    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg"
Oct 31 00:46:06.244050 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:06.240616    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/191c12a0-fb78-4586-8d87-80f3cf3dce22-metrics-certs\" (UniqueName: \"kubernetes.io/secret/191c12a0-fb78-4586-8d87-80f3cf3dce22-metrics-certs\") pod \"191c12a0-fb78-4586-8d87-80f3cf3dce22\" (UID: \"191c12a0-fb78-4586-8d87-80f3cf3dce22\") "
Oct 31 00:46:06.244050 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:06.240768    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/191c12a0-fb78-4586-8d87-80f3cf3dce22-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/191c12a0-fb78-4586-8d87-80f3cf3dce22-openshift-service-ca\") pod \"191c12a0-fb78-4586-8d87-80f3cf3dce22\" (UID: \"191c12a0-fb78-4586-8d87-80f3cf3dce22\") "
Oct 31 00:46:06.244050 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:06.240870    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/191c12a0-fb78-4586-8d87-80f3cf3dce22-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/191c12a0-fb78-4586-8d87-80f3cf3dce22-maas-api-tls\") pod \"191c12a0-fb78-4586-8d87-80f3cf3dce22\" (UID: \"191c12a0-fb78-4586-8d87-80f3cf3dce22\") "
Oct 31 00:46:06.244050 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:06.240955    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/191c12a0-fb78-4586-8d87-80f3cf3dce22-kube-api-access-jgf2s\" (UniqueName: \"kubernetes.io/projected/191c12a0-fb78-4586-8d87-80f3cf3dce22-kube-api-access-jgf2s\") pod \"191c12a0-fb78-4586-8d87-80f3cf3dce22\" (UID: \"191c12a0-fb78-4586-8d87-80f3cf3dce22\") "
Oct 31 00:46:06.244050 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:06.243711    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191c12a0-fb78-4586-8d87-80f3cf3dce22-openshift-service-ca" pod "191c12a0-fb78-4586-8d87-80f3cf3dce22" (UID: "191c12a0-fb78-4586-8d87-80f3cf3dce22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:46:06.244802 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:06.244764    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/191c12a0-fb78-4586-8d87-80f3cf3dce22-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:46:06.251100 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:06.251053    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/191c12a0-fb78-4586-8d87-80f3cf3dce22-metrics-certs" pod "191c12a0-fb78-4586-8d87-80f3cf3dce22" (UID: "191c12a0-fb78-4586-8d87-80f3cf3dce22"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:46:06.251888 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:06.251847    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/191c12a0-fb78-4586-8d87-80f3cf3dce22-maas-api-tls" pod "191c12a0-fb78-4586-8d87-80f3cf3dce22" (UID: "191c12a0-fb78-4586-8d87-80f3cf3dce22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:46:06.252348 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:06.252311    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191c12a0-fb78-4586-8d87-80f3cf3dce22-kube-api-access-jgf2s" pod "191c12a0-fb78-4586-8d87-80f3cf3dce22" (UID: "191c12a0-fb78-4586-8d87-80f3cf3dce22"). InnerVolumeSpecName "kube-api-access-jgf2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:46:06.391713 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:06.378654    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/191c12a0-fb78-4586-8d87-80f3cf3dce22-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:46:06.391713 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:06.378713    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/191c12a0-fb78-4586-8d87-80f3cf3dce22-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:46:06.391713 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:06.378729    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgf2s\" (UniqueName: \"kubernetes.io/projected/191c12a0-fb78-4586-8d87-80f3cf3dce22-kube-api-access-jgf2s\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:46:07.235306 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:07.235269    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg" event={"ID":"191c12a0-fb78-4586-8d87-80f3cf3dce22","Type":"ContainerDied","Data":"a188901a0645c752f0d600600e77dac7bf6217363d3aad9d7a6d21ed21facda0"}
Oct 31 00:46:07.235839 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:07.235341    4327 scope.go:122] "RemoveContainer" containerID="3311a8a035a6f8553a080ac0db77b89b5e167b96ac82c9f022ac9e400979003a"
Oct 31 00:46:07.235839 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:07.235529    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg"
Oct 31 00:46:07.291496 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:07.291454    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg"]
Oct 31 00:46:07.303578 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:07.303548    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b08f569a-77b5f55dd4-fbtvg"]
Oct 31 00:46:07.542168 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:07.542071    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"]
Oct 31 00:46:07.542845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:07.542811    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd" podUID="ff2dc9c9-da34-4576-b759-8f1f6b632e51" containerName="istio-proxy" containerID="cri-o://4eb25bf775888fe94a95f96a52dfd4a8037da2ab69019898c857f66ca9dbb23d" gracePeriod=30
Oct 31 00:46:08.075650 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:08.075604    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="191c12a0-fb78-4586-8d87-80f3cf3dce22" path="/var/lib/kubelet/pods/191c12a0-fb78-4586-8d87-80f3cf3dce22/volumes"
Oct 31 00:46:10.437941 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.437910    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:46:10.596545 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.596453    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istiod-ca-cert\") pod \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") "
Oct 31 00:46:10.596749 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.596591    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-data\") pod \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") "
Oct 31 00:46:10.596749 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.596708    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-credential-socket\") pod \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") "
Oct 31 00:46:10.596749 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.596744    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-workload-certs\") pod \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") "
Oct 31 00:46:10.596922 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.596855    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-token\" (UniqueName: \"kubernetes.io/projected/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-token\") pod \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") "
Oct 31 00:46:10.596922 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.596887    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-data" pod "ff2dc9c9-da34-4576-b759-8f1f6b632e51" (UID: "ff2dc9c9-da34-4576-b759-8f1f6b632e51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:46:10.597027 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.596918    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff2dc9c9-da34-4576-b759-8f1f6b632e51-kube-api-access-lkr4r\" (UniqueName: \"kubernetes.io/projected/ff2dc9c9-da34-4576-b759-8f1f6b632e51-kube-api-access-lkr4r\") pod \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") "
Oct 31 00:46:10.597027 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.596941    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istiod-ca-cert" pod "ff2dc9c9-da34-4576-b759-8f1f6b632e51" (UID: "ff2dc9c9-da34-4576-b759-8f1f6b632e51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:46:10.597027 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.596994    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-podinfo\") pod \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") "
Oct 31 00:46:10.597171 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.597021    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-credential-socket" pod "ff2dc9c9-da34-4576-b759-8f1f6b632e51" (UID: "ff2dc9c9-da34-4576-b759-8f1f6b632e51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:46:10.597171 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.597053    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-workload-certs" pod "ff2dc9c9-da34-4576-b759-8f1f6b632e51" (UID: "ff2dc9c9-da34-4576-b759-8f1f6b632e51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:46:10.597171 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.597065    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-workload-socket\") pod \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") "
Oct 31 00:46:10.597171 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.597159    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-envoy\") pod \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\" (UID: \"ff2dc9c9-da34-4576-b759-8f1f6b632e51\") "
Oct 31 00:46:10.597358 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.597258    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-workload-socket" pod "ff2dc9c9-da34-4576-b759-8f1f6b632e51" (UID: "ff2dc9c9-da34-4576-b759-8f1f6b632e51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:46:10.597812 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.597792    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:46:10.597914 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.597816    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:46:10.597914 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.597831    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:46:10.597914 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.597843    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:46:10.597914 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.597856    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:46:10.599228 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.599207    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-podinfo" pod "ff2dc9c9-da34-4576-b759-8f1f6b632e51" (UID: "ff2dc9c9-da34-4576-b759-8f1f6b632e51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:46:10.599228 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.599222    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-envoy" pod "ff2dc9c9-da34-4576-b759-8f1f6b632e51" (UID: "ff2dc9c9-da34-4576-b759-8f1f6b632e51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:46:10.599465 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.599448    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-token" pod "ff2dc9c9-da34-4576-b759-8f1f6b632e51" (UID: "ff2dc9c9-da34-4576-b759-8f1f6b632e51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:46:10.599504 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.599482    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff2dc9c9-da34-4576-b759-8f1f6b632e51-kube-api-access-lkr4r" pod "ff2dc9c9-da34-4576-b759-8f1f6b632e51" (UID: "ff2dc9c9-da34-4576-b759-8f1f6b632e51"). InnerVolumeSpecName "kube-api-access-lkr4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:46:10.699297 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.699260    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:46:10.699297 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.699291    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkr4r\" (UniqueName: \"kubernetes.io/projected/ff2dc9c9-da34-4576-b759-8f1f6b632e51-kube-api-access-lkr4r\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:46:10.699297 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.699301    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:46:10.699592 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:10.699310    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff2dc9c9-da34-4576-b759-8f1f6b632e51-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:46:11.266902 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:11.266866    4327 generic.go:356] "Generic (PLEG): container finished" podID="ff2dc9c9-da34-4576-b759-8f1f6b632e51" containerID="4eb25bf775888fe94a95f96a52dfd4a8037da2ab69019898c857f66ca9dbb23d" exitCode=0
Oct 31 00:46:11.267097 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:11.266929    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"
Oct 31 00:46:11.267097 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:11.266948    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd" event={"ID":"ff2dc9c9-da34-4576-b759-8f1f6b632e51","Type":"ContainerDied","Data":"4eb25bf775888fe94a95f96a52dfd4a8037da2ab69019898c857f66ca9dbb23d"}
Oct 31 00:46:11.267097 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:11.267013    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd" event={"ID":"ff2dc9c9-da34-4576-b759-8f1f6b632e51","Type":"ContainerDied","Data":"196dfc374197b206a540a69f96d3b81ddefd6d5a9fd6ae96402c9e5b79450cac"}
Oct 31 00:46:11.267097 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:11.267045    4327 scope.go:122] "RemoveContainer" containerID="4eb25bf775888fe94a95f96a52dfd4a8037da2ab69019898c857f66ca9dbb23d"
Oct 31 00:46:11.279362 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:11.279340    4327 scope.go:122] "RemoveContainer" containerID="4eb25bf775888fe94a95f96a52dfd4a8037da2ab69019898c857f66ca9dbb23d"
Oct 31 00:46:11.279637 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:46:11.279620    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eb25bf775888fe94a95f96a52dfd4a8037da2ab69019898c857f66ca9dbb23d\": container with ID starting with 4eb25bf775888fe94a95f96a52dfd4a8037da2ab69019898c857f66ca9dbb23d not found: ID does not exist" containerID="4eb25bf775888fe94a95f96a52dfd4a8037da2ab69019898c857f66ca9dbb23d"
Oct 31 00:46:11.279710 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:11.279648    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eb25bf775888fe94a95f96a52dfd4a8037da2ab69019898c857f66ca9dbb23d"} err="failed to get container status \"4eb25bf775888fe94a95f96a52dfd4a8037da2ab69019898c857f66ca9dbb23d\": rpc error: code = NotFound desc = could not find container \"4eb25bf775888fe94a95f96a52dfd4a8037da2ab69019898c857f66ca9dbb23d\": container with ID starting with 4eb25bf775888fe94a95f96a52dfd4a8037da2ab69019898c857f66ca9dbb23d not found: ID does not exist"
Oct 31 00:46:11.335336 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:11.335306    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"]
Oct 31 00:46:11.348420 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:11.348379    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b08f569a-openshift-default-5fd5c4857f-4nlvd"]
Oct 31 00:46:11.922469 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:11.922434    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff2dc9c9-da34-4576-b759-8f1f6b632e51" path="/var/lib/kubelet/pods/ff2dc9c9-da34-4576-b759-8f1f6b632e51/volumes"
Oct 31 00:46:14.987048 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:14.987019    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b"
Oct 31 00:46:15.301895 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:46:15.301805    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:46:16.188762 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.188734    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb"
Oct 31 00:46:16.285079 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.285045    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"]
Oct 31 00:46:16.286161 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.286138    4327 cpu_manager.go:414] "containerMap: removing container" podUID="ff2dc9c9-da34-4576-b759-8f1f6b632e51" containerName="istio-proxy"
Oct 31 00:46:16.286235 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.286166    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff2dc9c9-da34-4576-b759-8f1f6b632e51" containerName="istio-proxy"
Oct 31 00:46:16.286235 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.286202    4327 cpu_manager.go:414] "containerMap: removing container" podUID="191c12a0-fb78-4586-8d87-80f3cf3dce22" containerName="maas-api"
Oct 31 00:46:16.286235 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.286210    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="191c12a0-fb78-4586-8d87-80f3cf3dce22" containerName="maas-api"
Oct 31 00:46:16.286424 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.286407    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff2dc9c9-da34-4576-b759-8f1f6b632e51" containerName="istio-proxy"
Oct 31 00:46:16.286476 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.286432    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="191c12a0-fb78-4586-8d87-80f3cf3dce22" containerName="maas-api"
Oct 31 00:46:16.291344 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.291319    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.312237 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.312210    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-498580-openshift-default-dockercfg-8p4s8\""
Oct 31 00:46:16.318561 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.318410    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"]
Oct 31 00:46:16.442407 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.433286    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-workload-certs\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.442407 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.433333    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9brl\" (UniqueName: \"kubernetes.io/projected/d965e342-4052-4f57-a495-4d8023a1f882-kube-api-access-w9brl\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.442407 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.433391    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d965e342-4052-4f57-a495-4d8023a1f882-istio-token\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.442407 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.433419    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d965e342-4052-4f57-a495-4d8023a1f882-istio-podinfo\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.442407 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.433554    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-istio-data\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.442407 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.433802    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-istio-envoy\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.442407 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.433977    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-credential-socket\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.442407 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.434008    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d965e342-4052-4f57-a495-4d8023a1f882-istiod-ca-cert\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.442407 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.434111    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-workload-socket\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.558061 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.558018    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-istio-data\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.698461 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.698372    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-istio-envoy\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.698878 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.698853    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-credential-socket\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.699035 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.699018    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d965e342-4052-4f57-a495-4d8023a1f882-istiod-ca-cert\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.699223 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.699207    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-workload-socket\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.699429 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.699414    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-workload-certs\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.699556 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.699541    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9brl\" (UniqueName: \"kubernetes.io/projected/d965e342-4052-4f57-a495-4d8023a1f882-kube-api-access-w9brl\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.699702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.699671    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d965e342-4052-4f57-a495-4d8023a1f882-istio-token\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.699821 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.699808    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d965e342-4052-4f57-a495-4d8023a1f882-istio-podinfo\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.707714 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.703361    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d965e342-4052-4f57-a495-4d8023a1f882-istiod-ca-cert\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.707714 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.572387    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-istio-data\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.707714 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.704299    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-workload-certs\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.707714 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.704574    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-workload-socket\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.707714 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.704868    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-credential-socket\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.745269 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.745228    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-istio-envoy\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.751334 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.751130    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d965e342-4052-4f57-a495-4d8023a1f882-istio-podinfo\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.778724 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.765385    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d965e342-4052-4f57-a495-4d8023a1f882-istio-token\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.788716 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.787423    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9brl\" (UniqueName: \"kubernetes.io/projected/d965e342-4052-4f57-a495-4d8023a1f882-kube-api-access-w9brl\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:16.920705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:16.915433    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:17.348168 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:17.348142    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"]
Oct 31 00:46:17.349137 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:46:17.349110    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd965e342_4052_4f57_a495_4d8023a1f882.slice/crio-6b362cad5b33ac96f757b07b0dacfeda577465ef3f08128bd1c30e3f2ae7b7df WatchSource:0}: Error finding container 6b362cad5b33ac96f757b07b0dacfeda577465ef3f08128bd1c30e3f2ae7b7df: Status 404 returned error can't find the container with id 6b362cad5b33ac96f757b07b0dacfeda577465ef3f08128bd1c30e3f2ae7b7df
Oct 31 00:46:17.351868 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:17.351835    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:46:17.351934 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:17.351901    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:46:17.351934 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:17.351930    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:46:18.336986 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:18.336946    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj" event={"ID":"d965e342-4052-4f57-a495-4d8023a1f882","Type":"ContainerStarted","Data":"07c1a71556991ba0e4182aecd33df8c8bbb24eca38740efa9f61c6afc21a50b6"}
Oct 31 00:46:18.337145 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:18.337016    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj" event={"ID":"d965e342-4052-4f57-a495-4d8023a1f882","Type":"ContainerStarted","Data":"6b362cad5b33ac96f757b07b0dacfeda577465ef3f08128bd1c30e3f2ae7b7df"}
Oct 31 00:46:18.403486 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:18.403426    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj" podStartSLOduration=2.403411138 podStartE2EDuration="2.403411138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:18.397789904 +0000 UTC m=+1638.619842821" watchObservedRunningTime="2026-10-31 00:46:18.403411138 +0000 UTC m=+1638.625446683"
Oct 31 00:46:18.923184 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:18.923145    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:18.929095 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:18.929067    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:19.340739 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:19.340634    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t"]
Oct 31 00:46:19.344063 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:19.344027    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t"
Oct 31 00:46:19.344657 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:19.344605    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:19.345654 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:19.345633    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:46:19.361324 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:19.361276    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-36af31-serving-cert\""
Oct 31 00:46:19.368223 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:19.368197    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t"]
Oct 31 00:46:19.480500 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:19.480456    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t\" (UID: \"0c0783ab-8c5d-4d27-9073-e2a9bc8ede71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t"
Oct 31 00:46:19.481437 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:19.481405    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t\" (UID: \"0c0783ab-8c5d-4d27-9073-e2a9bc8ede71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t"
Oct 31 00:46:19.482912 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:19.482881    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qjn8\" (UniqueName: \"kubernetes.io/projected/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-kube-api-access-9qjn8\") pod \"maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t\" (UID: \"0c0783ab-8c5d-4d27-9073-e2a9bc8ede71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t"
Oct 31 00:46:19.483846 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:19.482974    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t\" (UID: \"0c0783ab-8c5d-4d27-9073-e2a9bc8ede71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t"
Oct 31 00:46:19.584597 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:19.584564    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t\" (UID: \"0c0783ab-8c5d-4d27-9073-e2a9bc8ede71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t"
Oct 31 00:46:19.584873 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:19.584781    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qjn8\" (UniqueName: \"kubernetes.io/projected/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-kube-api-access-9qjn8\") pod \"maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t\" (UID: \"0c0783ab-8c5d-4d27-9073-e2a9bc8ede71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t"
Oct 31 00:46:19.584873 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:19.584843    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t\" (UID: \"0c0783ab-8c5d-4d27-9073-e2a9bc8ede71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t"
Oct 31 00:46:19.584993 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:19.584977    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t\" (UID: \"0c0783ab-8c5d-4d27-9073-e2a9bc8ede71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t"
Oct 31 00:46:19.585652 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:19.585627    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t\" (UID: \"0c0783ab-8c5d-4d27-9073-e2a9bc8ede71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t"
Oct 31 00:46:19.587393 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:19.587367    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t\" (UID: \"0c0783ab-8c5d-4d27-9073-e2a9bc8ede71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t"
Oct 31 00:46:19.587540 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:19.587522    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t\" (UID: \"0c0783ab-8c5d-4d27-9073-e2a9bc8ede71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t"
Oct 31 00:46:19.608857 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:19.608822    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qjn8\" (UniqueName: \"kubernetes.io/projected/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-kube-api-access-9qjn8\") pod \"maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t\" (UID: \"0c0783ab-8c5d-4d27-9073-e2a9bc8ede71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t"
Oct 31 00:46:19.684711 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:19.668812    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t"
Oct 31 00:46:20.172838 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:20.172805    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t"]
Oct 31 00:46:20.173579 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:46:20.173553    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0783ab_8c5d_4d27_9073_e2a9bc8ede71.slice/crio-179dfae242c99cf8b55eb99ef156e0a41d4b5e40c427b5ed2138c271f5705af1 WatchSource:0}: Error finding container 179dfae242c99cf8b55eb99ef156e0a41d4b5e40c427b5ed2138c271f5705af1: Status 404 returned error can't find the container with id 179dfae242c99cf8b55eb99ef156e0a41d4b5e40c427b5ed2138c271f5705af1
Oct 31 00:46:20.359470 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:20.359429    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t" event={"ID":"0c0783ab-8c5d-4d27-9073-e2a9bc8ede71","Type":"ContainerStarted","Data":"179dfae242c99cf8b55eb99ef156e0a41d4b5e40c427b5ed2138c271f5705af1"}
Oct 31 00:46:21.410082 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:21.406817    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t" event={"ID":"0c0783ab-8c5d-4d27-9073-e2a9bc8ede71","Type":"ContainerStarted","Data":"481c251a423fc07fc30ac8bea402a191de8975cbc6a28b6945b2b9e05e12a790"}
Oct 31 00:46:21.410082 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:21.407550    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t"
Oct 31 00:46:21.461185 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:21.461111    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t" podStartSLOduration=2.108101446 podStartE2EDuration="2.461089432s" totalImagesPullingTime="352.987986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:21.460085892 +0000 UTC m=+1641.682121436" watchObservedRunningTime="2026-10-31 00:46:21.461089432 +0000 UTC m=+1641.683124974"
Oct 31 00:46:22.943539 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:22.943505    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8"]
Oct 31 00:46:22.946930 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:22.946912    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8"
Oct 31 00:46:22.975069 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:22.975039    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-498580\""
Oct 31 00:46:22.975069 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:22.975039    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-498580-dockercfg-2nnbf\""
Oct 31 00:46:22.980537 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:22.980508    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8"]
Oct 31 00:46:23.113962 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:23.107538    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9949e8e5-8f11-4c06-8a7b-20e8640c2ac6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8\" (UID: \"9949e8e5-8f11-4c06-8a7b-20e8640c2ac6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8"
Oct 31 00:46:23.113962 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:23.107583    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjlpr\" (UniqueName: \"kubernetes.io/projected/9949e8e5-8f11-4c06-8a7b-20e8640c2ac6-kube-api-access-mjlpr\") pod \"payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8\" (UID: \"9949e8e5-8f11-4c06-8a7b-20e8640c2ac6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8"
Oct 31 00:46:23.252755 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:23.250113    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9949e8e5-8f11-4c06-8a7b-20e8640c2ac6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8\" (UID: \"9949e8e5-8f11-4c06-8a7b-20e8640c2ac6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8"
Oct 31 00:46:23.252755 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:23.250170    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjlpr\" (UniqueName: \"kubernetes.io/projected/9949e8e5-8f11-4c06-8a7b-20e8640c2ac6-kube-api-access-mjlpr\") pod \"payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8\" (UID: \"9949e8e5-8f11-4c06-8a7b-20e8640c2ac6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8"
Oct 31 00:46:23.252755 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:23.251772    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9949e8e5-8f11-4c06-8a7b-20e8640c2ac6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8\" (UID: \"9949e8e5-8f11-4c06-8a7b-20e8640c2ac6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8"
Oct 31 00:46:23.318257 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:23.318216    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjlpr\" (UniqueName: \"kubernetes.io/projected/9949e8e5-8f11-4c06-8a7b-20e8640c2ac6-kube-api-access-mjlpr\") pod \"payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8\" (UID: \"9949e8e5-8f11-4c06-8a7b-20e8640c2ac6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8"
Oct 31 00:46:23.348315 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:23.346200    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c"]
Oct 31 00:46:23.463672 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:23.463631    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c"
Oct 31 00:46:23.502738 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:23.491711    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c"]
Oct 31 00:46:23.564101 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:23.564064    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8"
Oct 31 00:46:23.605110 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:23.605079    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fe10361-91bc-4ff6-a6d4-faec973ad665-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c\" (UID: \"5fe10361-91bc-4ff6-a6d4-faec973ad665\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c"
Oct 31 00:46:23.605269 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:23.605197    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbh4v\" (UniqueName: \"kubernetes.io/projected/5fe10361-91bc-4ff6-a6d4-faec973ad665-kube-api-access-tbh4v\") pod \"payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c\" (UID: \"5fe10361-91bc-4ff6-a6d4-faec973ad665\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c"
Oct 31 00:46:23.706899 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:23.706866    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fe10361-91bc-4ff6-a6d4-faec973ad665-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c\" (UID: \"5fe10361-91bc-4ff6-a6d4-faec973ad665\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c"
Oct 31 00:46:23.707103 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:23.706945    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbh4v\" (UniqueName: \"kubernetes.io/projected/5fe10361-91bc-4ff6-a6d4-faec973ad665-kube-api-access-tbh4v\") pod \"payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c\" (UID: \"5fe10361-91bc-4ff6-a6d4-faec973ad665\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c"
Oct 31 00:46:23.707665 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:23.707638    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fe10361-91bc-4ff6-a6d4-faec973ad665-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c\" (UID: \"5fe10361-91bc-4ff6-a6d4-faec973ad665\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c"
Oct 31 00:46:23.733549 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:23.733520    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbh4v\" (UniqueName: \"kubernetes.io/projected/5fe10361-91bc-4ff6-a6d4-faec973ad665-kube-api-access-tbh4v\") pod \"payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c\" (UID: \"5fe10361-91bc-4ff6-a6d4-faec973ad665\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c"
Oct 31 00:46:23.826456 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:23.826377    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c"
Oct 31 00:46:23.873891 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:23.873854    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8"]
Oct 31 00:46:23.874258 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:46:23.874233    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9949e8e5_8f11_4c06_8a7b_20e8640c2ac6.slice/crio-1e3ae62ddd339686c51c1d6543977c655d08b004a719c70dedbe4f16fe45d253 WatchSource:0}: Error finding container 1e3ae62ddd339686c51c1d6543977c655d08b004a719c70dedbe4f16fe45d253: Status 404 returned error can't find the container with id 1e3ae62ddd339686c51c1d6543977c655d08b004a719c70dedbe4f16fe45d253
Oct 31 00:46:24.277704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:24.232050    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c"]
Oct 31 00:46:24.556622 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:24.556426    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c" event={"ID":"5fe10361-91bc-4ff6-a6d4-faec973ad665","Type":"ContainerStarted","Data":"11c2bebd9e9fc900f3695b62eb103594eadde2b7978f29a7d666970c04a5192c"}
Oct 31 00:46:24.560011 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:24.559642    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8" event={"ID":"9949e8e5-8f11-4c06-8a7b-20e8640c2ac6","Type":"ContainerStarted","Data":"74253eb04a8953575096c0b5814519e61e35839ac1ee27fc56b57774965ddac1"}
Oct 31 00:46:24.560011 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:24.559723    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8" event={"ID":"9949e8e5-8f11-4c06-8a7b-20e8640c2ac6","Type":"ContainerStarted","Data":"1e3ae62ddd339686c51c1d6543977c655d08b004a719c70dedbe4f16fe45d253"}
Oct 31 00:46:24.561624 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:24.561383    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8"
Oct 31 00:46:24.606151 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:24.606096    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8" podStartSLOduration=2.606080711 podStartE2EDuration="2.606080711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:24.60278854 +0000 UTC m=+1644.824824082" watchObservedRunningTime="2026-10-31 00:46:24.606080711 +0000 UTC m=+1644.828116271"
Oct 31 00:46:25.568352 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:25.568308    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c" event={"ID":"5fe10361-91bc-4ff6-a6d4-faec973ad665","Type":"ContainerStarted","Data":"62b842af61f45b8bdbd3df0a55ffc33bd6cf81d6390eb05636e8904a575db817"}
Oct 31 00:46:25.568789 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:25.568605    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c"
Oct 31 00:46:25.610776 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:25.610713    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c" podStartSLOduration=3.610667462 podStartE2EDuration="3.610667462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:25.61035974 +0000 UTC m=+1645.832395283" watchObservedRunningTime="2026-10-31 00:46:25.610667462 +0000 UTC m=+1645.832703003"
Oct 31 00:46:27.419582 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:27.419554    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t"
Oct 31 00:46:36.579790 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:36.579759    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8"
Oct 31 00:46:36.580191 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:36.579844    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c"
Oct 31 00:46:42.583756 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:42.583445    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c2cd8283-587685db69-76crd"]
Oct 31 00:46:42.615117 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:42.615078    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz"]
Oct 31 00:46:45.052650 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.052614    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"]
Oct 31 00:46:45.058617 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.058563    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.068168 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.068104    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"]
Oct 31 00:46:45.089742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.082194    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-f2bdf3-openshift-default-dockercfg-xzqfh\""
Oct 31 00:46:45.089742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.087250    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-data\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.089742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.087371    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-token\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.089742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.087428    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-credential-socket\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.089742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.087470    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb6f2\" (UniqueName: \"kubernetes.io/projected/b38a6a95-545c-43c6-a861-645d17ddfab4-kube-api-access-rb6f2\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.089742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.087501    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-envoy\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.089742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.087590    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-podinfo\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.089742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.087654    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-workload-certs\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.089742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.087711    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38a6a95-545c-43c6-a861-645d17ddfab4-istiod-ca-cert\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.089742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.087775    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-workload-socket\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.192703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.190773    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-credential-socket\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.192703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.190843    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb6f2\" (UniqueName: \"kubernetes.io/projected/b38a6a95-545c-43c6-a861-645d17ddfab4-kube-api-access-rb6f2\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.192703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.190876    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-envoy\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.192703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.190975    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-podinfo\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.192703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.191039    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-workload-certs\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.192703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.191076    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38a6a95-545c-43c6-a861-645d17ddfab4-istiod-ca-cert\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.192703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.191130    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-workload-socket\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.192703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.191311    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-data\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.192703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.191429    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-token\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.192703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.192256    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-credential-socket\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.241739 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.238381    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38a6a95-545c-43c6-a861-645d17ddfab4-istiod-ca-cert\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.241739 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.240105    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-workload-certs\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.241739 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.240363    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-workload-socket\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.241739 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.240610    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-data\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.440703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.438369    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-token\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.458731 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.448095    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-envoy\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.673242 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.655217    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-podinfo\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.673242 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.670142    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb6f2\" (UniqueName: \"kubernetes.io/projected/b38a6a95-545c-43c6-a861-645d17ddfab4-kube-api-access-rb6f2\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:45.754657 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:45.754574    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:46.418410 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:46.418361    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 00:46:46.475700 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:46.472776    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:46:46.665702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:46.664970    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:46:46.765754 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:46:46.749319    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:46:47.598310 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:47.598264    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-c2cd8283-587685db69-76crd" podUID="cd485e06-0892-45a9-9419-3f410e9f866d" containerName="payload-processing" containerID="cri-o://14933828e84238d521da847a7ef6dfe942e112ec825f79fd6d1c76586bf01cf0" gracePeriod=25
Oct 31 00:46:47.616133 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:47.615918    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz" podUID="1c0c856c-f6b5-4d3c-ae26-84d887ee092e" containerName="payload-pre-processing" containerID="cri-o://a9efaebde7873a537cc942468879591f69addb55dfae75cd2f8ad45beb7380a8" gracePeriod=25
Oct 31 00:46:47.651467 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:47.651432    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"]
Oct 31 00:46:47.690016 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:46:47.689973    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38a6a95_545c_43c6_a861_645d17ddfab4.slice/crio-e53e4316201c515c1e181b9034fd2e9f8fa377a9e618cbcaa1402c909e6085d8 WatchSource:0}: Error finding container e53e4316201c515c1e181b9034fd2e9f8fa377a9e618cbcaa1402c909e6085d8: Status 404 returned error can't find the container with id e53e4316201c515c1e181b9034fd2e9f8fa377a9e618cbcaa1402c909e6085d8
Oct 31 00:46:47.753587 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:47.753555    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:46:47.753641 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:47.753630    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:46:47.753696 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:47.753661    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:46:48.005852 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.005819    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c2cd8283-587685db69-76crd"
Oct 31 00:46:48.060321 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.060280    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd485e06-0892-45a9-9419-3f410e9f866d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd485e06-0892-45a9-9419-3f410e9f866d-plugins-config-volume\") pod \"cd485e06-0892-45a9-9419-3f410e9f866d\" (UID: \"cd485e06-0892-45a9-9419-3f410e9f866d\") "
Oct 31 00:46:48.060909 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.060644    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd485e06-0892-45a9-9419-3f410e9f866d-kube-api-access-2ndcn\" (UniqueName: \"kubernetes.io/projected/cd485e06-0892-45a9-9419-3f410e9f866d-kube-api-access-2ndcn\") pod \"cd485e06-0892-45a9-9419-3f410e9f866d\" (UID: \"cd485e06-0892-45a9-9419-3f410e9f866d\") "
Oct 31 00:46:48.061455 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.061206    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd485e06-0892-45a9-9419-3f410e9f866d-plugins-config-volume" pod "cd485e06-0892-45a9-9419-3f410e9f866d" (UID: "cd485e06-0892-45a9-9419-3f410e9f866d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:46:48.061455 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.061415    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd485e06-0892-45a9-9419-3f410e9f866d-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:46:48.065501 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.065457    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd485e06-0892-45a9-9419-3f410e9f866d-kube-api-access-2ndcn" pod "cd485e06-0892-45a9-9419-3f410e9f866d" (UID: "cd485e06-0892-45a9-9419-3f410e9f866d"). InnerVolumeSpecName "kube-api-access-2ndcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:46:48.087250 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.087220    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz"
Oct 31 00:46:48.163047 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.162952    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c0c856c-f6b5-4d3c-ae26-84d887ee092e-kube-api-access-z6szv\" (UniqueName: \"kubernetes.io/projected/1c0c856c-f6b5-4d3c-ae26-84d887ee092e-kube-api-access-z6szv\") pod \"1c0c856c-f6b5-4d3c-ae26-84d887ee092e\" (UID: \"1c0c856c-f6b5-4d3c-ae26-84d887ee092e\") "
Oct 31 00:46:48.163224 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.163098    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c0c856c-f6b5-4d3c-ae26-84d887ee092e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0c856c-f6b5-4d3c-ae26-84d887ee092e-plugins-config-volume\") pod \"1c0c856c-f6b5-4d3c-ae26-84d887ee092e\" (UID: \"1c0c856c-f6b5-4d3c-ae26-84d887ee092e\") "
Oct 31 00:46:48.163619 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.163598    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ndcn\" (UniqueName: \"kubernetes.io/projected/cd485e06-0892-45a9-9419-3f410e9f866d-kube-api-access-2ndcn\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:46:48.163860 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.163833    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c0c856c-f6b5-4d3c-ae26-84d887ee092e-plugins-config-volume" pod "1c0c856c-f6b5-4d3c-ae26-84d887ee092e" (UID: "1c0c856c-f6b5-4d3c-ae26-84d887ee092e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:46:48.166003 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.165972    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0c856c-f6b5-4d3c-ae26-84d887ee092e-kube-api-access-z6szv" pod "1c0c856c-f6b5-4d3c-ae26-84d887ee092e" (UID: "1c0c856c-f6b5-4d3c-ae26-84d887ee092e"). InnerVolumeSpecName "kube-api-access-z6szv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:46:48.264966 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.264927    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6szv\" (UniqueName: \"kubernetes.io/projected/1c0c856c-f6b5-4d3c-ae26-84d887ee092e-kube-api-access-z6szv\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:46:48.264966 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.264958    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0c856c-f6b5-4d3c-ae26-84d887ee092e-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:46:48.460708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.460594    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv" event={"ID":"b38a6a95-545c-43c6-a861-645d17ddfab4","Type":"ContainerStarted","Data":"66b8ef3bdab3fc9045e444361fb0143be82cf98130331ecd99b03ccb08a4c247"}
Oct 31 00:46:48.460893 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.460708    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv" event={"ID":"b38a6a95-545c-43c6-a861-645d17ddfab4","Type":"ContainerStarted","Data":"e53e4316201c515c1e181b9034fd2e9f8fa377a9e618cbcaa1402c909e6085d8"}
Oct 31 00:46:48.462155 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.462127    4327 generic.go:356] "Generic (PLEG): container finished" podID="cd485e06-0892-45a9-9419-3f410e9f866d" containerID="14933828e84238d521da847a7ef6dfe942e112ec825f79fd6d1c76586bf01cf0" exitCode=0
Oct 31 00:46:48.462302 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.462182    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c2cd8283-587685db69-76crd"
Oct 31 00:46:48.462302 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.462220    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c2cd8283-587685db69-76crd" event={"ID":"cd485e06-0892-45a9-9419-3f410e9f866d","Type":"ContainerDied","Data":"14933828e84238d521da847a7ef6dfe942e112ec825f79fd6d1c76586bf01cf0"}
Oct 31 00:46:48.462302 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.462290    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c2cd8283-587685db69-76crd" event={"ID":"cd485e06-0892-45a9-9419-3f410e9f866d","Type":"ContainerDied","Data":"25f9516bf0c5d6f6dd6b6c2c34c4bd81a55d22504999e878836bc036ee88cdc8"}
Oct 31 00:46:48.462470 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.462325    4327 scope.go:122] "RemoveContainer" containerID="14933828e84238d521da847a7ef6dfe942e112ec825f79fd6d1c76586bf01cf0"
Oct 31 00:46:48.463884 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.463857    4327 generic.go:356] "Generic (PLEG): container finished" podID="1c0c856c-f6b5-4d3c-ae26-84d887ee092e" containerID="a9efaebde7873a537cc942468879591f69addb55dfae75cd2f8ad45beb7380a8" exitCode=0
Oct 31 00:46:48.463978 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.463925    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz"
Oct 31 00:46:48.464067 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.464042    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz" event={"ID":"1c0c856c-f6b5-4d3c-ae26-84d887ee092e","Type":"ContainerDied","Data":"a9efaebde7873a537cc942468879591f69addb55dfae75cd2f8ad45beb7380a8"}
Oct 31 00:46:48.464154 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.464097    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz" event={"ID":"1c0c856c-f6b5-4d3c-ae26-84d887ee092e","Type":"ContainerDied","Data":"3641f93f84482551492bd83b3c1172facd096a9c4c18f49be27319daa5ef2055"}
Oct 31 00:46:48.481216 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.481163    4327 scope.go:122] "RemoveContainer" containerID="14933828e84238d521da847a7ef6dfe942e112ec825f79fd6d1c76586bf01cf0"
Oct 31 00:46:48.482093 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:46:48.482066    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14933828e84238d521da847a7ef6dfe942e112ec825f79fd6d1c76586bf01cf0\": container with ID starting with 14933828e84238d521da847a7ef6dfe942e112ec825f79fd6d1c76586bf01cf0 not found: ID does not exist" containerID="14933828e84238d521da847a7ef6dfe942e112ec825f79fd6d1c76586bf01cf0"
Oct 31 00:46:48.482196 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.482102    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14933828e84238d521da847a7ef6dfe942e112ec825f79fd6d1c76586bf01cf0"} err="failed to get container status \"14933828e84238d521da847a7ef6dfe942e112ec825f79fd6d1c76586bf01cf0\": rpc error: code = NotFound desc = could not find container \"14933828e84238d521da847a7ef6dfe942e112ec825f79fd6d1c76586bf01cf0\": container with ID starting with 14933828e84238d521da847a7ef6dfe942e112ec825f79fd6d1c76586bf01cf0 not found: ID does not exist"
Oct 31 00:46:48.482196 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.482125    4327 scope.go:122] "RemoveContainer" containerID="a9efaebde7873a537cc942468879591f69addb55dfae75cd2f8ad45beb7380a8"
Oct 31 00:46:48.491914 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.491889    4327 scope.go:122] "RemoveContainer" containerID="a9efaebde7873a537cc942468879591f69addb55dfae75cd2f8ad45beb7380a8"
Oct 31 00:46:48.492207 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:46:48.492185    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9efaebde7873a537cc942468879591f69addb55dfae75cd2f8ad45beb7380a8\": container with ID starting with a9efaebde7873a537cc942468879591f69addb55dfae75cd2f8ad45beb7380a8 not found: ID does not exist" containerID="a9efaebde7873a537cc942468879591f69addb55dfae75cd2f8ad45beb7380a8"
Oct 31 00:46:48.492266 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.492217    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9efaebde7873a537cc942468879591f69addb55dfae75cd2f8ad45beb7380a8"} err="failed to get container status \"a9efaebde7873a537cc942468879591f69addb55dfae75cd2f8ad45beb7380a8\": rpc error: code = NotFound desc = could not find container \"a9efaebde7873a537cc942468879591f69addb55dfae75cd2f8ad45beb7380a8\": container with ID starting with a9efaebde7873a537cc942468879591f69addb55dfae75cd2f8ad45beb7380a8 not found: ID does not exist"
Oct 31 00:46:48.511711 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.511646    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv" podStartSLOduration=4.511631629 podStartE2EDuration="4.511631629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:48.510314344 +0000 UTC m=+1668.732349885" watchObservedRunningTime="2026-10-31 00:46:48.511631629 +0000 UTC m=+1668.733667174"
Oct 31 00:46:48.568083 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.568046    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz"]
Oct 31 00:46:48.579942 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.579908    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c2cd8283-546b74d947-5bjpz"]
Oct 31 00:46:48.615228 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.615198    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c2cd8283-587685db69-76crd"]
Oct 31 00:46:48.692837 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.692798    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c2cd8283-587685db69-76crd"]
Oct 31 00:46:48.775790 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.768360    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:48.791703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:48.789239    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:49.475057 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:49.475006    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:49.476126 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:49.476104    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:46:49.922405 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:49.922373    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c0c856c-f6b5-4d3c-ae26-84d887ee092e" path="/var/lib/kubelet/pods/1c0c856c-f6b5-4d3c-ae26-84d887ee092e/volumes"
Oct 31 00:46:49.922937 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:49.922922    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd485e06-0892-45a9-9419-3f410e9f866d" path="/var/lib/kubelet/pods/cd485e06-0892-45a9-9419-3f410e9f866d/volumes"
Oct 31 00:46:54.743295 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:54.743259    4327 generic.go:356] "Generic (PLEG): container finished" podID="004c3b2b-bdd7-42df-a961-3ce60631a8b4" containerID="c1c99c775dd03e98676e576977294ef0847c4e7acccbb1df88db67cf093a330e" exitCode=0
Oct 31 00:46:54.743755 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:54.743333    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp" event={"ID":"004c3b2b-bdd7-42df-a961-3ce60631a8b4","Type":"ContainerDied","Data":"c1c99c775dd03e98676e576977294ef0847c4e7acccbb1df88db67cf093a330e"}
Oct 31 00:46:54.743932 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:54.743915    4327 scope.go:122] "RemoveContainer" containerID="c1c99c775dd03e98676e576977294ef0847c4e7acccbb1df88db67cf093a330e"
Oct 31 00:46:54.745745 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:54.745715    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:46:54.745823 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:54.745788    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:46:54.745862 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:54.745824    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:46:55.754640 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:55.754601    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp" event={"ID":"004c3b2b-bdd7-42df-a961-3ce60631a8b4","Type":"ContainerStarted","Data":"fd9b6b017c1ec9da5fa9f6a07a015f2f0f0a1291610550f0d516d1708bf8a7e2"}
Oct 31 00:46:56.424794 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:56.424759    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:46:56.443254 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:56.443217    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 00:46:56.443535 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:56.443505    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp" podUID="004c3b2b-bdd7-42df-a961-3ce60631a8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 00:46:56.985129 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:56.985098    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:46:57.423746 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:57.421541    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 00:46:57.423746 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:57.421611    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp" podUID="004c3b2b-bdd7-42df-a961-3ce60631a8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 00:46:58.441960 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:58.441913    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 00:46:58.442454 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:58.441992    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp" podUID="004c3b2b-bdd7-42df-a961-3ce60631a8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 00:46:59.441721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:59.438144    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 00:46:59.441721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:46:59.438213    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp" podUID="004c3b2b-bdd7-42df-a961-3ce60631a8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 00:47:00.432184 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:00.432144    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 00:47:00.432872 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:00.432840    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp" podUID="004c3b2b-bdd7-42df-a961-3ce60631a8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 00:47:01.460977 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:01.460943    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:47:01.463793 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:01.463762    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:47:04.956703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:04.955787    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 00:47:05.046698 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:05.035067    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:47:05.415703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:05.413794    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:47:05.415703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:05.414422    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:47:05.643729 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:05.628832    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:47:05.658951 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:05.657858    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:47:05.824769 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:05.823766    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:47:05.956915 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:05.956833    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:47:05.964022 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:05.963992    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:47:06.035762 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:06.035713    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:47:06.052460 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:06.052430    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:47:06.410974 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:06.410624    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 00:47:06.410974 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:06.410724    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 00:47:13.504815 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:13.504771    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk"]
Oct 31 00:47:13.520705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:13.517898    4327 cpu_manager.go:414] "containerMap: removing container" podUID="cd485e06-0892-45a9-9419-3f410e9f866d" containerName="payload-processing"
Oct 31 00:47:13.520957 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:13.520930    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd485e06-0892-45a9-9419-3f410e9f866d" containerName="payload-processing"
Oct 31 00:47:13.542744 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:13.542709    4327 cpu_manager.go:414] "containerMap: removing container" podUID="1c0c856c-f6b5-4d3c-ae26-84d887ee092e" containerName="payload-pre-processing"
Oct 31 00:47:13.542982 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:13.542960    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c0c856c-f6b5-4d3c-ae26-84d887ee092e" containerName="payload-pre-processing"
Oct 31 00:47:13.549890 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:13.543352    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c0c856c-f6b5-4d3c-ae26-84d887ee092e" containerName="payload-pre-processing"
Oct 31 00:47:13.550157 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:13.550133    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd485e06-0892-45a9-9419-3f410e9f866d" containerName="payload-processing"
Oct 31 00:47:13.762406 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:13.762320    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk"
Oct 31 00:47:13.779728 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:13.762192    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt"]
Oct 31 00:47:13.797756 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:13.793540    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkjzc\" (UniqueName: \"kubernetes.io/projected/67146ff0-b14c-4bb9-9a6b-db437c984bbd-kube-api-access-wkjzc\") pod \"payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk\" (UID: \"67146ff0-b14c-4bb9-9a6b-db437c984bbd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk"
Oct 31 00:47:13.798506 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:13.798460    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67146ff0-b14c-4bb9-9a6b-db437c984bbd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk\" (UID: \"67146ff0-b14c-4bb9-9a6b-db437c984bbd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk"
Oct 31 00:47:13.798973 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:13.798950    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-3c90d8\""
Oct 31 00:47:13.877695 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:13.877652    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt"
Oct 31 00:47:13.878638 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:13.878609    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk"]
Oct 31 00:47:13.878785 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:13.878704    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt"]
Oct 31 00:47:13.900116 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:13.900087    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b302fecf-fc39-4122-bc35-83585cc68ae5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt\" (UID: \"b302fecf-fc39-4122-bc35-83585cc68ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt"
Oct 31 00:47:13.900448 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:13.900428    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67146ff0-b14c-4bb9-9a6b-db437c984bbd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk\" (UID: \"67146ff0-b14c-4bb9-9a6b-db437c984bbd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk"
Oct 31 00:47:13.900671 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:13.900651    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnmk2\" (UniqueName: \"kubernetes.io/projected/b302fecf-fc39-4122-bc35-83585cc68ae5-kube-api-access-bnmk2\") pod \"payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt\" (UID: \"b302fecf-fc39-4122-bc35-83585cc68ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt"
Oct 31 00:47:13.900913 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:13.900894    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkjzc\" (UniqueName: \"kubernetes.io/projected/67146ff0-b14c-4bb9-9a6b-db437c984bbd-kube-api-access-wkjzc\") pod \"payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk\" (UID: \"67146ff0-b14c-4bb9-9a6b-db437c984bbd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk"
Oct 31 00:47:13.901305 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:13.901277    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67146ff0-b14c-4bb9-9a6b-db437c984bbd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk\" (UID: \"67146ff0-b14c-4bb9-9a6b-db437c984bbd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk"
Oct 31 00:47:13.940542 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:13.940510    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkjzc\" (UniqueName: \"kubernetes.io/projected/67146ff0-b14c-4bb9-9a6b-db437c984bbd-kube-api-access-wkjzc\") pod \"payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk\" (UID: \"67146ff0-b14c-4bb9-9a6b-db437c984bbd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk"
Oct 31 00:47:14.002401 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:14.002364    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnmk2\" (UniqueName: \"kubernetes.io/projected/b302fecf-fc39-4122-bc35-83585cc68ae5-kube-api-access-bnmk2\") pod \"payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt\" (UID: \"b302fecf-fc39-4122-bc35-83585cc68ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt"
Oct 31 00:47:14.002656 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:14.002634    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b302fecf-fc39-4122-bc35-83585cc68ae5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt\" (UID: \"b302fecf-fc39-4122-bc35-83585cc68ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt"
Oct 31 00:47:14.003307 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:14.003281    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b302fecf-fc39-4122-bc35-83585cc68ae5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt\" (UID: \"b302fecf-fc39-4122-bc35-83585cc68ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt"
Oct 31 00:47:14.028056 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:14.027978    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnmk2\" (UniqueName: \"kubernetes.io/projected/b302fecf-fc39-4122-bc35-83585cc68ae5-kube-api-access-bnmk2\") pod \"payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt\" (UID: \"b302fecf-fc39-4122-bc35-83585cc68ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt"
Oct 31 00:47:14.178135 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:14.178102    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk"
Oct 31 00:47:14.198244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:14.198210    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt"
Oct 31 00:47:14.569796 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:14.569762    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk"]
Oct 31 00:47:14.572665 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:47:14.572637    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67146ff0_b14c_4bb9_9a6b_db437c984bbd.slice/crio-94eebf8eb4faf02a8022c7597c5867b7d3e216a5ab843b79d29d465af827d5c5 WatchSource:0}: Error finding container 94eebf8eb4faf02a8022c7597c5867b7d3e216a5ab843b79d29d465af827d5c5: Status 404 returned error can't find the container with id 94eebf8eb4faf02a8022c7597c5867b7d3e216a5ab843b79d29d465af827d5c5
Oct 31 00:47:14.713335 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:47:14.713300    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb302fecf_fc39_4122_bc35_83585cc68ae5.slice/crio-559f3fb132fdafcead3ff6e506fc524f66622a30ad64eccc45e8e4661f25678d WatchSource:0}: Error finding container 559f3fb132fdafcead3ff6e506fc524f66622a30ad64eccc45e8e4661f25678d: Status 404 returned error can't find the container with id 559f3fb132fdafcead3ff6e506fc524f66622a30ad64eccc45e8e4661f25678d
Oct 31 00:47:14.714939 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:14.714814    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt"]
Oct 31 00:47:15.295854 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:47:15.295811    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:47:15.484022 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:15.483980    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk" event={"ID":"67146ff0-b14c-4bb9-9a6b-db437c984bbd","Type":"ContainerStarted","Data":"cc12fda3ec16bcf761da60cd3a1fc701c6e66b4b97850d583f7b74d429edd4a5"}
Oct 31 00:47:15.484220 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:15.484079    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk"
Oct 31 00:47:15.484220 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:15.484099    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk" event={"ID":"67146ff0-b14c-4bb9-9a6b-db437c984bbd","Type":"ContainerStarted","Data":"94eebf8eb4faf02a8022c7597c5867b7d3e216a5ab843b79d29d465af827d5c5"}
Oct 31 00:47:15.485947 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:15.485920    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt" event={"ID":"b302fecf-fc39-4122-bc35-83585cc68ae5","Type":"ContainerStarted","Data":"e90f60efab88ef607254881b9b829967130a5a8c64e46ace7fb6dcbb0eabdc6f"}
Oct 31 00:47:15.486096 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:15.485967    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt" event={"ID":"b302fecf-fc39-4122-bc35-83585cc68ae5","Type":"ContainerStarted","Data":"559f3fb132fdafcead3ff6e506fc524f66622a30ad64eccc45e8e4661f25678d"}
Oct 31 00:47:15.486096 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:15.486009    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt"
Oct 31 00:47:15.527200 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:15.527138    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk" podStartSLOduration=2.527125228 podStartE2EDuration="2.527125228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:15.526205631 +0000 UTC m=+1695.748241173" watchObservedRunningTime="2026-10-31 00:47:15.527125228 +0000 UTC m=+1695.749160770"
Oct 31 00:47:15.574878 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:15.574747    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt" podStartSLOduration=2.574726838 podStartE2EDuration="2.574726838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:15.568453123 +0000 UTC m=+1695.790488665" watchObservedRunningTime="2026-10-31 00:47:15.574726838 +0000 UTC m=+1695.796762381"
Oct 31 00:47:16.500424 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:16.500344    4327 generic.go:356] "Generic (PLEG): container finished" podID="d965e342-4052-4f57-a495-4d8023a1f882" containerID="07c1a71556991ba0e4182aecd33df8c8bbb24eca38740efa9f61c6afc21a50b6" exitCode=0
Oct 31 00:47:16.500638 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:16.500424    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj" event={"ID":"d965e342-4052-4f57-a495-4d8023a1f882","Type":"ContainerDied","Data":"07c1a71556991ba0e4182aecd33df8c8bbb24eca38740efa9f61c6afc21a50b6"}
Oct 31 00:47:16.500978 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:16.500959    4327 scope.go:122] "RemoveContainer" containerID="07c1a71556991ba0e4182aecd33df8c8bbb24eca38740efa9f61c6afc21a50b6"
Oct 31 00:47:16.502596 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:16.502565    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:47:16.502657 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:16.502641    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:47:16.502731 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:16.502694    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:47:17.510111 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:17.510076    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj" event={"ID":"d965e342-4052-4f57-a495-4d8023a1f882","Type":"ContainerStarted","Data":"9e1d1d0f26ce648fa7c79e5d512dbcbef5f13af346249bebe31800975ac20c91"}
Oct 31 00:47:17.938702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:17.938415    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 00:47:17.938702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:17.938471    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj" podUID="d965e342-4052-4f57-a495-4d8023a1f882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 00:47:18.015705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:18.015020    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:47:18.941706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:18.938273    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 00:47:18.941706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:18.938362    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj" podUID="d965e342-4052-4f57-a495-4d8023a1f882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 00:47:19.345008 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:19.344918    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:47:19.966701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:19.939382    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 00:47:19.966701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:19.939540    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj" podUID="d965e342-4052-4f57-a495-4d8023a1f882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 00:47:20.938700 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:20.931089    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 00:47:20.938700 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:20.931153    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj" podUID="d965e342-4052-4f57-a495-4d8023a1f882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 00:47:21.940706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:21.936078    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 00:47:21.940706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:21.936143    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj" podUID="d965e342-4052-4f57-a495-4d8023a1f882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 00:47:22.923031 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:22.922994    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 00:47:22.923210 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:22.923052    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj" podUID="d965e342-4052-4f57-a495-4d8023a1f882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 00:47:23.926939 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:23.926910    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:47:23.928210 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:23.928185    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 00:47:26.540714 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:26.538248    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk"
Oct 31 00:47:26.556707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:26.554903    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt"
Oct 31 00:47:42.287713 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:42.256074    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"]
Oct 31 00:47:42.309087 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:42.308950    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"
Oct 31 00:47:42.342703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:42.317980    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"]
Oct 31 00:47:42.344463 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:42.344432    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-208e9f32\"/\"default-dockercfg-m7zh5\""
Oct 31 00:47:42.356798 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:42.356774    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-208e9f32\"/\"kube-root-ca.crt\""
Oct 31 00:47:42.356938 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:42.356829    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-208e9f32\"/\"openshift-service-ca.crt\""
Oct 31 00:47:42.357003 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:42.356909    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-208e9f32\"/\"e2e-lifecycle-model-208e9f32-kserve-self-signed-certs\""
Oct 31 00:47:42.481280 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:42.481236    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-model-cache\") pod \"e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz\" (UID: \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\") " pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"
Oct 31 00:47:42.481491 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:42.481388    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-tls-certs\") pod \"e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz\" (UID: \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\") " pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"
Oct 31 00:47:42.481550 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:42.481513    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-tmp-dir\") pod \"e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz\" (UID: \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\") " pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"
Oct 31 00:47:42.481603 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:42.481560    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-home\") pod \"e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz\" (UID: \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\") " pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"
Oct 31 00:47:42.481726 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:42.481687    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-dshm\") pod \"e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz\" (UID: \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\") " pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"
Oct 31 00:47:42.481858 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:42.481840    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cxhz\" (UniqueName: \"kubernetes.io/projected/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-kube-api-access-8cxhz\") pod \"e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz\" (UID: \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\") " pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"
Oct 31 00:47:42.589750 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:42.584578    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-tls-certs\") pod \"e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz\" (UID: \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\") " pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"
Oct 31 00:47:42.589750 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:42.584728    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-tmp-dir\") pod \"e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz\" (UID: \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\") " pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"
Oct 31 00:47:42.589750 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:42.584780    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-home\") pod \"e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz\" (UID: \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\") " pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"
Oct 31 00:47:42.589750 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:42.584854    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-dshm\") pod \"e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz\" (UID: \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\") " pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"
Oct 31 00:47:42.589750 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:42.584986    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cxhz\" (UniqueName: \"kubernetes.io/projected/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-kube-api-access-8cxhz\") pod \"e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz\" (UID: \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\") " pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"
Oct 31 00:47:42.589750 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:42.585149    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-model-cache\") pod \"e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz\" (UID: \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\") " pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"
Oct 31 00:47:42.589750 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:42.585770    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-home\") pod \"e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz\" (UID: \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\") " pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"
Oct 31 00:47:42.589750 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:42.586710    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-tmp-dir\") pod \"e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz\" (UID: \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\") " pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"
Oct 31 00:47:42.602767 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:42.591433    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-tls-certs\") pod \"e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz\" (UID: \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\") " pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"
Oct 31 00:47:42.604433 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:42.597873    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-model-cache\") pod \"e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz\" (UID: \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\") " pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"
Oct 31 00:47:42.604653 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:42.604348    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-dshm\") pod \"e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz\" (UID: \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\") " pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"
Oct 31 00:47:42.614350 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:42.614315    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cxhz\" (UniqueName: \"kubernetes.io/projected/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-kube-api-access-8cxhz\") pod \"e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz\" (UID: \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\") " pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"
Oct 31 00:47:42.655210 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:42.655177    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"
Oct 31 00:47:43.268849 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:47:43.268809    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1c60d59_fe16_4899_b8e4_e10fa5f7d9a6.slice/crio-da7c9d66addcdec157447a2115b6646414fa58c02ebe347b31e7648694fc4f47 WatchSource:0}: Error finding container da7c9d66addcdec157447a2115b6646414fa58c02ebe347b31e7648694fc4f47: Status 404 returned error can't find the container with id da7c9d66addcdec157447a2115b6646414fa58c02ebe347b31e7648694fc4f47
Oct 31 00:47:43.269177 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:43.269152    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"]
Oct 31 00:47:44.004148 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:44.004111    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz" event={"ID":"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6","Type":"ContainerStarted","Data":"f4d5e0f8eb34fbcb1d64a211c50cf91807c245020e0f2ddaf1a6a3e92f7d48a2"}
Oct 31 00:47:44.004605 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:44.004166    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz" event={"ID":"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6","Type":"ContainerStarted","Data":"da7c9d66addcdec157447a2115b6646414fa58c02ebe347b31e7648694fc4f47"}
Oct 31 00:47:44.050755 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:44.050670    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz" podStartSLOduration=2.050652553 podStartE2EDuration="2.050652553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:44.047024944 +0000 UTC m=+1724.269060487" watchObservedRunningTime="2026-10-31 00:47:44.050652553 +0000 UTC m=+1724.272688141"
Oct 31 00:47:44.642968 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:44.642931    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb"]
Oct 31 00:47:44.647951 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:44.647918    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb"
Oct 31 00:47:44.657289 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:44.657257    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"
Oct 31 00:47:44.664468 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:44.664224    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-bcdb4e-dockercfg-f5tns\""
Oct 31 00:47:44.668207 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:44.668182    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb"]
Oct 31 00:47:44.691593 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:44.691471    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-bcdb4e\""
Oct 31 00:47:44.792702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:44.781091    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9n55\" (UniqueName: \"kubernetes.io/projected/c975bedf-cccf-4102-bd06-d62598138f5d-kube-api-access-r9n55\") pod \"payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb\" (UID: \"c975bedf-cccf-4102-bd06-d62598138f5d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb"
Oct 31 00:47:44.792702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:44.781187    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c975bedf-cccf-4102-bd06-d62598138f5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb\" (UID: \"c975bedf-cccf-4102-bd06-d62598138f5d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb"
Oct 31 00:47:44.889702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:44.887937    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9n55\" (UniqueName: \"kubernetes.io/projected/c975bedf-cccf-4102-bd06-d62598138f5d-kube-api-access-r9n55\") pod \"payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb\" (UID: \"c975bedf-cccf-4102-bd06-d62598138f5d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb"
Oct 31 00:47:44.889702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:44.888025    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c975bedf-cccf-4102-bd06-d62598138f5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb\" (UID: \"c975bedf-cccf-4102-bd06-d62598138f5d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb"
Oct 31 00:47:44.889702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:44.889591    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c975bedf-cccf-4102-bd06-d62598138f5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb\" (UID: \"c975bedf-cccf-4102-bd06-d62598138f5d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb"
Oct 31 00:47:45.064303 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:45.056114    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9n55\" (UniqueName: \"kubernetes.io/projected/c975bedf-cccf-4102-bd06-d62598138f5d-kube-api-access-r9n55\") pod \"payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb\" (UID: \"c975bedf-cccf-4102-bd06-d62598138f5d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb"
Oct 31 00:47:45.288213 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:45.288112    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb"
Oct 31 00:47:45.477343 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:47:45.477308    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:47:45.770259 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:45.770227    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb"]
Oct 31 00:47:45.770492 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:47:45.770466    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc975bedf_cccf_4102_bd06_d62598138f5d.slice/crio-69de28b595792d6466496f6a3605f29ae7737d4e0ba36fb31bda9e227e86d977 WatchSource:0}: Error finding container 69de28b595792d6466496f6a3605f29ae7737d4e0ba36fb31bda9e227e86d977: Status 404 returned error can't find the container with id 69de28b595792d6466496f6a3605f29ae7737d4e0ba36fb31bda9e227e86d977
Oct 31 00:47:46.072519 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:46.072419    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb" event={"ID":"c975bedf-cccf-4102-bd06-d62598138f5d","Type":"ContainerStarted","Data":"d655d0ce9fdaefb1328609c8f15dd16f510b537562df15d3471d1a8808dcd63d"}
Oct 31 00:47:46.073003 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:46.072946    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb"
Oct 31 00:47:46.073069 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:46.073003    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb" event={"ID":"c975bedf-cccf-4102-bd06-d62598138f5d","Type":"ContainerStarted","Data":"69de28b595792d6466496f6a3605f29ae7737d4e0ba36fb31bda9e227e86d977"}
Oct 31 00:47:46.114378 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:46.114312    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb" podStartSLOduration=2.114296088 podStartE2EDuration="2.114296088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:46.112972635 +0000 UTC m=+1726.335008177" watchObservedRunningTime="2026-10-31 00:47:46.114296088 +0000 UTC m=+1726.336331647"
Oct 31 00:47:49.108737 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:49.107266    4327 generic.go:356] "Generic (PLEG): container finished" podID="b38a6a95-545c-43c6-a861-645d17ddfab4" containerID="66b8ef3bdab3fc9045e444361fb0143be82cf98130331ecd99b03ccb08a4c247" exitCode=0
Oct 31 00:47:49.108737 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:49.107376    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv" event={"ID":"b38a6a95-545c-43c6-a861-645d17ddfab4","Type":"ContainerDied","Data":"66b8ef3bdab3fc9045e444361fb0143be82cf98130331ecd99b03ccb08a4c247"}
Oct 31 00:47:49.108737 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:49.107990    4327 scope.go:122] "RemoveContainer" containerID="66b8ef3bdab3fc9045e444361fb0143be82cf98130331ecd99b03ccb08a4c247"
Oct 31 00:47:49.110949 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:49.110892    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:47:49.111125 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:49.111097    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:47:49.111228 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:49.111209    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:47:49.475927 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:49.475890    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:47:50.116134 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:50.116097    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv" event={"ID":"b38a6a95-545c-43c6-a861-645d17ddfab4","Type":"ContainerStarted","Data":"3f92227528bca418bd077c891ea98b1837153ab7f84661ad8064ac181ee0295b"}
Oct 31 00:47:50.763408 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:50.763371    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:47:50.764585 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:50.764558    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 31 00:47:50.764740 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:50.764619    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv" podUID="b38a6a95-545c-43c6-a861-645d17ddfab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 31 00:47:51.763495 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:51.763453    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 31 00:47:51.764041 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:51.763521    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv" podUID="b38a6a95-545c-43c6-a861-645d17ddfab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 31 00:47:52.655805 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:52.655736    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"
Oct 31 00:47:52.668295 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:52.668265    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"
Oct 31 00:47:52.763882 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:52.763851    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 31 00:47:52.764369 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:52.764344    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv" podUID="b38a6a95-545c-43c6-a861-645d17ddfab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 31 00:47:53.181496 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:53.181468    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"
Oct 31 00:47:53.763520 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:53.763479    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 31 00:47:53.763729 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:53.763548    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv" podUID="b38a6a95-545c-43c6-a861-645d17ddfab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 31 00:47:54.769348 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:54.769300    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 31 00:47:54.769976 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:54.769929    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv" podUID="b38a6a95-545c-43c6-a861-645d17ddfab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 31 00:47:55.763569 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:55.763529    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 31 00:47:55.763776 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:55.763599    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv" podUID="b38a6a95-545c-43c6-a861-645d17ddfab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 31 00:47:56.768919 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:56.768887    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:47:56.769391 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:56.769378    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:47:56.770228 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:56.770207    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 00:47:57.085581 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:47:57.085507    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb"
Oct 31 00:48:04.433817 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:04.433768    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:48:06.298580 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:06.298545    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4"]
Oct 31 00:48:06.302212 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:06.302194    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4"
Oct 31 00:48:06.323376 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:06.323350    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4"]
Oct 31 00:48:06.324818 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:06.324791    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-14e039\""
Oct 31 00:48:06.482349 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:06.482298    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kcc2\" (UniqueName: \"kubernetes.io/projected/ddeeaec6-cb86-42c3-9847-ee011c7de8eb-kube-api-access-4kcc2\") pod \"payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4\" (UID: \"ddeeaec6-cb86-42c3-9847-ee011c7de8eb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4"
Oct 31 00:48:06.482949 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:06.482919    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddeeaec6-cb86-42c3-9847-ee011c7de8eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4\" (UID: \"ddeeaec6-cb86-42c3-9847-ee011c7de8eb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4"
Oct 31 00:48:06.552819 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:06.519640    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5"]
Oct 31 00:48:06.602704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:06.591777    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddeeaec6-cb86-42c3-9847-ee011c7de8eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4\" (UID: \"ddeeaec6-cb86-42c3-9847-ee011c7de8eb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4"
Oct 31 00:48:06.616710 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:06.603220    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kcc2\" (UniqueName: \"kubernetes.io/projected/ddeeaec6-cb86-42c3-9847-ee011c7de8eb-kube-api-access-4kcc2\") pod \"payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4\" (UID: \"ddeeaec6-cb86-42c3-9847-ee011c7de8eb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4"
Oct 31 00:48:06.648703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:06.619388    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddeeaec6-cb86-42c3-9847-ee011c7de8eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4\" (UID: \"ddeeaec6-cb86-42c3-9847-ee011c7de8eb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4"
Oct 31 00:48:06.728953 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:06.721987    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5"]
Oct 31 00:48:06.728953 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:06.722186    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5"
Oct 31 00:48:06.801986 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:06.801955    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-14e039-dockercfg-gdp9h\""
Oct 31 00:48:06.850802 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:06.840187    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx6v9\" (UniqueName: \"kubernetes.io/projected/b1822005-c1d3-479b-abb5-0b0f0a4b0cd1-kube-api-access-vx6v9\") pod \"payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5\" (UID: \"b1822005-c1d3-479b-abb5-0b0f0a4b0cd1\") " pod="openshift-ingress/payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5"
Oct 31 00:48:06.851299 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:06.851263    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1822005-c1d3-479b-abb5-0b0f0a4b0cd1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5\" (UID: \"b1822005-c1d3-479b-abb5-0b0f0a4b0cd1\") " pod="openshift-ingress/payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5"
Oct 31 00:48:06.852476 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:06.852389    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kcc2\" (UniqueName: \"kubernetes.io/projected/ddeeaec6-cb86-42c3-9847-ee011c7de8eb-kube-api-access-4kcc2\") pod \"payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4\" (UID: \"ddeeaec6-cb86-42c3-9847-ee011c7de8eb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4"
Oct 31 00:48:06.922701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:06.922196    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4"
Oct 31 00:48:06.979886 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:06.979851    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1822005-c1d3-479b-abb5-0b0f0a4b0cd1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5\" (UID: \"b1822005-c1d3-479b-abb5-0b0f0a4b0cd1\") " pod="openshift-ingress/payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5"
Oct 31 00:48:06.981151 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:06.981114    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1822005-c1d3-479b-abb5-0b0f0a4b0cd1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5\" (UID: \"b1822005-c1d3-479b-abb5-0b0f0a4b0cd1\") " pod="openshift-ingress/payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5"
Oct 31 00:48:06.981436 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:06.981414    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx6v9\" (UniqueName: \"kubernetes.io/projected/b1822005-c1d3-479b-abb5-0b0f0a4b0cd1-kube-api-access-vx6v9\") pod \"payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5\" (UID: \"b1822005-c1d3-479b-abb5-0b0f0a4b0cd1\") " pod="openshift-ingress/payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5"
Oct 31 00:48:07.006348 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:07.006295    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx6v9\" (UniqueName: \"kubernetes.io/projected/b1822005-c1d3-479b-abb5-0b0f0a4b0cd1-kube-api-access-vx6v9\") pod \"payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5\" (UID: \"b1822005-c1d3-479b-abb5-0b0f0a4b0cd1\") " pod="openshift-ingress/payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5"
Oct 31 00:48:07.162603 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:07.162567    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5"
Oct 31 00:48:07.306587 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:07.306558    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4"]
Oct 31 00:48:07.307056 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:48:07.307029    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddeeaec6_cb86_42c3_9847_ee011c7de8eb.slice/crio-22878b09ab1c48308ac1c8a0a918b2cc6a29f889aa0bb90eb6446d46b19c6380 WatchSource:0}: Error finding container 22878b09ab1c48308ac1c8a0a918b2cc6a29f889aa0bb90eb6446d46b19c6380: Status 404 returned error can't find the container with id 22878b09ab1c48308ac1c8a0a918b2cc6a29f889aa0bb90eb6446d46b19c6380
Oct 31 00:48:07.471734 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:07.471699    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5"]
Oct 31 00:48:07.472816 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:48:07.472781    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1822005_c1d3_479b_abb5_0b0f0a4b0cd1.slice/crio-2bc7ceabe0c8f9112e8944025e67a63d8053a1ef8de9ca040e0659af2fb0b253 WatchSource:0}: Error finding container 2bc7ceabe0c8f9112e8944025e67a63d8053a1ef8de9ca040e0659af2fb0b253: Status 404 returned error can't find the container with id 2bc7ceabe0c8f9112e8944025e67a63d8053a1ef8de9ca040e0659af2fb0b253
Oct 31 00:48:07.530570 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:07.530539    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4" event={"ID":"ddeeaec6-cb86-42c3-9847-ee011c7de8eb","Type":"ContainerStarted","Data":"88faae9fb66f887e93ee3edefc710e4e8fd5135309faf14595c4d0282cc7079d"}
Oct 31 00:48:07.530758 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:07.530592    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4" event={"ID":"ddeeaec6-cb86-42c3-9847-ee011c7de8eb","Type":"ContainerStarted","Data":"22878b09ab1c48308ac1c8a0a918b2cc6a29f889aa0bb90eb6446d46b19c6380"}
Oct 31 00:48:07.530758 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:07.530633    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4"
Oct 31 00:48:07.532403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:07.532383    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5" event={"ID":"b1822005-c1d3-479b-abb5-0b0f0a4b0cd1","Type":"ContainerStarted","Data":"2bc7ceabe0c8f9112e8944025e67a63d8053a1ef8de9ca040e0659af2fb0b253"}
Oct 31 00:48:07.575144 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:07.575080    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4" podStartSLOduration=1.575066055 podStartE2EDuration="1.575066055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:07.573334885 +0000 UTC m=+1747.795370427" watchObservedRunningTime="2026-10-31 00:48:07.575066055 +0000 UTC m=+1747.797101597"
Oct 31 00:48:08.545725 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:08.545654    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5" event={"ID":"b1822005-c1d3-479b-abb5-0b0f0a4b0cd1","Type":"ContainerStarted","Data":"eee15371f50606cf697279bd22df73dba21b52dc03af3c127bbbf9c958aedc75"}
Oct 31 00:48:08.546151 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:08.545915    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5"
Oct 31 00:48:08.594668 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:08.594610    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5" podStartSLOduration=2.5945938330000002 podStartE2EDuration="2.594593833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:08.592340472 +0000 UTC m=+1748.814375997" watchObservedRunningTime="2026-10-31 00:48:08.594593833 +0000 UTC m=+1748.816629379"
Oct 31 00:48:09.757344 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:09.757306    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf"]
Oct 31 00:48:09.761035 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:09.761014    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf"
Oct 31 00:48:09.781025 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:09.780995    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-bcdb4e-serving-cert\""
Oct 31 00:48:09.783987 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:09.783952    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf\" (UID: \"b7f71dd1-1b1e-4c68-8a2d-8641ea516faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf"
Oct 31 00:48:09.784831 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:09.784269    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf\" (UID: \"b7f71dd1-1b1e-4c68-8a2d-8641ea516faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf"
Oct 31 00:48:09.784831 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:09.784321    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggpmd\" (UniqueName: \"kubernetes.io/projected/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-kube-api-access-ggpmd\") pod \"maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf\" (UID: \"b7f71dd1-1b1e-4c68-8a2d-8641ea516faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf"
Oct 31 00:48:09.784831 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:09.784523    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf\" (UID: \"b7f71dd1-1b1e-4c68-8a2d-8641ea516faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf"
Oct 31 00:48:09.803987 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:09.803954    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf"]
Oct 31 00:48:09.889139 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:09.889105    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf\" (UID: \"b7f71dd1-1b1e-4c68-8a2d-8641ea516faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf"
Oct 31 00:48:09.889525 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:09.889503    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf\" (UID: \"b7f71dd1-1b1e-4c68-8a2d-8641ea516faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf"
Oct 31 00:48:09.890229 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:09.890198    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf\" (UID: \"b7f71dd1-1b1e-4c68-8a2d-8641ea516faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf"
Oct 31 00:48:09.890662 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:09.890633    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf\" (UID: \"b7f71dd1-1b1e-4c68-8a2d-8641ea516faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf"
Oct 31 00:48:09.890817 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:09.890727    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggpmd\" (UniqueName: \"kubernetes.io/projected/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-kube-api-access-ggpmd\") pod \"maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf\" (UID: \"b7f71dd1-1b1e-4c68-8a2d-8641ea516faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf"
Oct 31 00:48:09.910608 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:09.894581    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"]
Oct 31 00:48:09.910608 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:09.895475    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf\" (UID: \"b7f71dd1-1b1e-4c68-8a2d-8641ea516faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf"
Oct 31 00:48:09.910608 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:09.899917    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:09.910608 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:09.909404    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf\" (UID: \"b7f71dd1-1b1e-4c68-8a2d-8641ea516faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf"
Oct 31 00:48:09.915603 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:09.915572    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggpmd\" (UniqueName: \"kubernetes.io/projected/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-kube-api-access-ggpmd\") pod \"maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf\" (UID: \"b7f71dd1-1b1e-4c68-8a2d-8641ea516faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf"
Oct 31 00:48:09.919282 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:09.919254    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e166ca-openshift-default-dockercfg-z2svw\""
Oct 31 00:48:09.929856 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:09.929826    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"]
Oct 31 00:48:10.003702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.002635    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-credential-socket\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.003702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.002791    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-workload-certs\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.003702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.002877    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-envoy\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.003702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.002974    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-token\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.003702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.003118    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-workload-socket\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.003702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.003257    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-podinfo\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.003702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.003326    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-data\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.003702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.003431    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca0da87-ad0a-45cc-a022-d4946e7fab22-istiod-ca-cert\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.003702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.003613    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdcck\" (UniqueName: \"kubernetes.io/projected/aca0da87-ad0a-45cc-a022-d4946e7fab22-kube-api-access-hdcck\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.102858 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.102012    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf"
Oct 31 00:48:10.122977 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.120715    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-workload-socket\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.122977 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.120827    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-podinfo\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.122977 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.120874    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-data\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.122977 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.120943    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca0da87-ad0a-45cc-a022-d4946e7fab22-istiod-ca-cert\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.122977 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.121062    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdcck\" (UniqueName: \"kubernetes.io/projected/aca0da87-ad0a-45cc-a022-d4946e7fab22-kube-api-access-hdcck\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.122977 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.121205    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-credential-socket\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.122977 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.121314    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-workload-certs\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.122977 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.121396    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-envoy\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.122977 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.121493    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-token\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.128257 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.125797    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-workload-socket\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.179710 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.129294    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-data\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.179710 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.129425    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-workload-certs\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.179710 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.129671    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-credential-socket\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.179710 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.129951    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca0da87-ad0a-45cc-a022-d4946e7fab22-istiod-ca-cert\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.179710 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.145934    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-envoy\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.179710 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.167265    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-podinfo\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.234496 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.233477    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-token\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.289155 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.289083    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdcck\" (UniqueName: \"kubernetes.io/projected/aca0da87-ad0a-45cc-a022-d4946e7fab22-kube-api-access-hdcck\") pod \"e2e-api-a-e166ca-openshift-default-6584656577-7cxfh\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") " pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.562139 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.562056    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:10.856526 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.856498    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf"]
Oct 31 00:48:10.857013 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:48:10.856947    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f71dd1_1b1e_4c68_8a2d_8641ea516faf.slice/crio-9f35269253a82fed8123a82bf5d4ba0565be019264a5e7efbe2d534d427df297 WatchSource:0}: Error finding container 9f35269253a82fed8123a82bf5d4ba0565be019264a5e7efbe2d534d427df297: Status 404 returned error can't find the container with id 9f35269253a82fed8123a82bf5d4ba0565be019264a5e7efbe2d534d427df297
Oct 31 00:48:10.909167 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:48:10.909131    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca0da87_ad0a_45cc_a022_d4946e7fab22.slice/crio-bf7e8a8f8afe8f11ec978ea32a11ecdd00838eb269ce3096b816d21ae606ad8b WatchSource:0}: Error finding container bf7e8a8f8afe8f11ec978ea32a11ecdd00838eb269ce3096b816d21ae606ad8b: Status 404 returned error can't find the container with id bf7e8a8f8afe8f11ec978ea32a11ecdd00838eb269ce3096b816d21ae606ad8b
Oct 31 00:48:10.910329 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.910302    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"]
Oct 31 00:48:10.912513 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.912483    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:48:10.912590 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.912543    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:48:10.912590 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:10.912576    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:48:11.582000 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:11.581913    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh" event={"ID":"aca0da87-ad0a-45cc-a022-d4946e7fab22","Type":"ContainerStarted","Data":"2997c24959fba21be98fced109a4b85993b2dfc6151f35d7bc4c237aa3180251"}
Oct 31 00:48:11.582000 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:11.581977    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh" event={"ID":"aca0da87-ad0a-45cc-a022-d4946e7fab22","Type":"ContainerStarted","Data":"bf7e8a8f8afe8f11ec978ea32a11ecdd00838eb269ce3096b816d21ae606ad8b"}
Oct 31 00:48:11.583905 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:11.583872    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf" event={"ID":"b7f71dd1-1b1e-4c68-8a2d-8641ea516faf","Type":"ContainerStarted","Data":"e384579678c4de3cba6d8d7156a19866de0baa603a8e37ce0da881755c7ff7e7"}
Oct 31 00:48:11.584058 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:11.583926    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf" event={"ID":"b7f71dd1-1b1e-4c68-8a2d-8641ea516faf","Type":"ContainerStarted","Data":"9f35269253a82fed8123a82bf5d4ba0565be019264a5e7efbe2d534d427df297"}
Oct 31 00:48:11.584058 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:11.583953    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf"
Oct 31 00:48:11.646512 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:11.646443    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh" podStartSLOduration=2.646423452 podStartE2EDuration="2.646423452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:11.637922241 +0000 UTC m=+1751.859957786" watchObservedRunningTime="2026-10-31 00:48:11.646423452 +0000 UTC m=+1751.868458997"
Oct 31 00:48:11.687292 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:11.687177    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf" podStartSLOduration=2.234707006 podStartE2EDuration="2.687158388s" totalImagesPullingTime="452.451382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:11.685673538 +0000 UTC m=+1751.907709098" watchObservedRunningTime="2026-10-31 00:48:11.687158388 +0000 UTC m=+1751.909193942"
Oct 31 00:48:12.564409 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:12.564377    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:12.570593 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:12.570565    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:12.591382 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:12.591354    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:12.592246 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:12.592225    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:48:15.295266 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:48:15.295228    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:48:17.597254 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:17.597224    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf"
Oct 31 00:48:18.551491 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:18.551461    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4"
Oct 31 00:48:19.560423 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:19.560396    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5"
Oct 31 00:48:19.793362 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:19.793326    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm"]
Oct 31 00:48:19.798233 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:19.798212    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm"
Oct 31 00:48:19.821854 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:19.821555    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fa2c15c1\""
Oct 31 00:48:19.821854 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:19.821806    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-fa2c15c1-dockercfg-2x8zr\""
Oct 31 00:48:19.827401 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:19.827372    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm"]
Oct 31 00:48:19.945705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:19.945035    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19b21c31-d906-4652-81e1-765d08fc3e5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm\" (UID: \"19b21c31-d906-4652-81e1-765d08fc3e5e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm"
Oct 31 00:48:19.945705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:19.945113    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5vxg\" (UniqueName: \"kubernetes.io/projected/19b21c31-d906-4652-81e1-765d08fc3e5e-kube-api-access-p5vxg\") pod \"payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm\" (UID: \"19b21c31-d906-4652-81e1-765d08fc3e5e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm"
Oct 31 00:48:20.068749 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:20.062741    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19b21c31-d906-4652-81e1-765d08fc3e5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm\" (UID: \"19b21c31-d906-4652-81e1-765d08fc3e5e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm"
Oct 31 00:48:20.068749 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:20.062831    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5vxg\" (UniqueName: \"kubernetes.io/projected/19b21c31-d906-4652-81e1-765d08fc3e5e-kube-api-access-p5vxg\") pod \"payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm\" (UID: \"19b21c31-d906-4652-81e1-765d08fc3e5e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm"
Oct 31 00:48:20.068749 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:20.064669    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19b21c31-d906-4652-81e1-765d08fc3e5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm\" (UID: \"19b21c31-d906-4652-81e1-765d08fc3e5e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm"
Oct 31 00:48:20.236720 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:20.235590    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5vxg\" (UniqueName: \"kubernetes.io/projected/19b21c31-d906-4652-81e1-765d08fc3e5e-kube-api-access-p5vxg\") pod \"payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm\" (UID: \"19b21c31-d906-4652-81e1-765d08fc3e5e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm"
Oct 31 00:48:20.245824 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:20.245773    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq"]
Oct 31 00:48:20.252143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:20.252117    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq"
Oct 31 00:48:20.270703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:20.265026    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq"]
Oct 31 00:48:20.414965 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:20.413713    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm"
Oct 31 00:48:20.474629 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:20.474590    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzrz7\" (UniqueName: \"kubernetes.io/projected/9f9893fc-b75a-4bed-925f-c02168ffc76e-kube-api-access-wzrz7\") pod \"payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq\" (UID: \"9f9893fc-b75a-4bed-925f-c02168ffc76e\") " pod="openshift-ingress/payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq"
Oct 31 00:48:20.475062 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:20.475039    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f9893fc-b75a-4bed-925f-c02168ffc76e-plugins-config-volume\") pod \"payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq\" (UID: \"9f9893fc-b75a-4bed-925f-c02168ffc76e\") " pod="openshift-ingress/payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq"
Oct 31 00:48:20.586434 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:20.586392    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f9893fc-b75a-4bed-925f-c02168ffc76e-plugins-config-volume\") pod \"payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq\" (UID: \"9f9893fc-b75a-4bed-925f-c02168ffc76e\") " pod="openshift-ingress/payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq"
Oct 31 00:48:20.586988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:20.586485    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzrz7\" (UniqueName: \"kubernetes.io/projected/9f9893fc-b75a-4bed-925f-c02168ffc76e-kube-api-access-wzrz7\") pod \"payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq\" (UID: \"9f9893fc-b75a-4bed-925f-c02168ffc76e\") " pod="openshift-ingress/payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq"
Oct 31 00:48:20.589918 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:20.589891    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f9893fc-b75a-4bed-925f-c02168ffc76e-plugins-config-volume\") pod \"payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq\" (UID: \"9f9893fc-b75a-4bed-925f-c02168ffc76e\") " pod="openshift-ingress/payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq"
Oct 31 00:48:20.614106 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:20.614069    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzrz7\" (UniqueName: \"kubernetes.io/projected/9f9893fc-b75a-4bed-925f-c02168ffc76e-kube-api-access-wzrz7\") pod \"payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq\" (UID: \"9f9893fc-b75a-4bed-925f-c02168ffc76e\") " pod="openshift-ingress/payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq"
Oct 31 00:48:20.691909 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:20.691868    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq"
Oct 31 00:48:20.766376 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:20.766342    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm"]
Oct 31 00:48:21.034581 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:21.034554    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq"]
Oct 31 00:48:21.254790 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:21.254751    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm" event={"ID":"19b21c31-d906-4652-81e1-765d08fc3e5e","Type":"ContainerStarted","Data":"5f85f06de98a67b708f185f3df346b475c9242e2ff724c3dd41a20ca4982b9ef"}
Oct 31 00:48:21.254961 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:21.254819    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm" event={"ID":"19b21c31-d906-4652-81e1-765d08fc3e5e","Type":"ContainerStarted","Data":"bc5507cb7e15fc048c497a455024bb145a80d5ce3ca67b86c3b2dea72d3f6980"}
Oct 31 00:48:21.255271 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:21.255251    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm"
Oct 31 00:48:21.256665 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:21.256644    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq" event={"ID":"9f9893fc-b75a-4bed-925f-c02168ffc76e","Type":"ContainerStarted","Data":"ead9d4cbb81ca58e22d70c3854a6714287e3d59d5ae07275a99ea41853dad039"}
Oct 31 00:48:21.256791 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:21.256744    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq"
Oct 31 00:48:21.256791 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:21.256776    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq" event={"ID":"9f9893fc-b75a-4bed-925f-c02168ffc76e","Type":"ContainerStarted","Data":"a2df8bde484d53edd68822215944296713033ae5e3eb01a77c23829d038281dd"}
Oct 31 00:48:21.299277 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:21.299165    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm" podStartSLOduration=2.299145823 podStartE2EDuration="2.299145823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:21.296222265 +0000 UTC m=+1761.518257832" watchObservedRunningTime="2026-10-31 00:48:21.299145823 +0000 UTC m=+1761.521181367"
Oct 31 00:48:21.344121 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:21.344047    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq" podStartSLOduration=2.344027852 podStartE2EDuration="2.344027852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:21.341612816 +0000 UTC m=+1761.563648357" watchObservedRunningTime="2026-10-31 00:48:21.344027852 +0000 UTC m=+1761.566063395"
Oct 31 00:48:32.267295 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:32.267261    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm"
Oct 31 00:48:32.267789 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:32.267743    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq"
Oct 31 00:48:43.819215 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:43.819160    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"]
Oct 31 00:48:43.854215 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:43.849609    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"]
Oct 31 00:48:43.854215 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:43.849752    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:43.892973 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:43.892533    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-714f8f-openshift-default-dockercfg-687wt\""
Oct 31 00:48:44.178093 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.064941    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-workload-socket\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:44.178093 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.065018    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-istio-data\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:44.178093 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.065082    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-credential-socket\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:44.178093 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.065242    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-istio-envoy\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:44.178093 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.065426    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a811bb1a-42b1-4184-b123-c295547116d2-istiod-ca-cert\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:44.178093 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.065507    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a811bb1a-42b1-4184-b123-c295547116d2-istio-podinfo\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:44.178093 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.065555    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-workload-certs\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:44.178093 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.065613    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhpdx\" (UniqueName: \"kubernetes.io/projected/a811bb1a-42b1-4184-b123-c295547116d2-kube-api-access-hhpdx\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:44.178093 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.065644    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a811bb1a-42b1-4184-b123-c295547116d2-istio-token\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:44.206729 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.203451    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a811bb1a-42b1-4184-b123-c295547116d2-istio-token\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:44.206729 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.203576    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-workload-socket\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:44.206729 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.203644    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-istio-data\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:44.206729 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.203754    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-credential-socket\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:44.206729 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.204711    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-credential-socket\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:44.213143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.212591    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-workload-socket\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:44.213143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.212965    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-istio-data\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:44.332702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.331945    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhpdx\" (UniqueName: \"kubernetes.io/projected/a811bb1a-42b1-4184-b123-c295547116d2-kube-api-access-hhpdx\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:44.332702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.332223    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-istio-envoy\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:44.332702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.332387    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a811bb1a-42b1-4184-b123-c295547116d2-istiod-ca-cert\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:44.332702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.332465    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a811bb1a-42b1-4184-b123-c295547116d2-istio-podinfo\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:44.332702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.332515    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-workload-certs\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:44.488759 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.398344    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a811bb1a-42b1-4184-b123-c295547116d2-istiod-ca-cert\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:44.488759 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.439045    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a811bb1a-42b1-4184-b123-c295547116d2-istio-podinfo\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:44.635157 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.634665    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-workload-certs\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:44.747311 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.747181    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a811bb1a-42b1-4184-b123-c295547116d2-istio-token\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:44.836603 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.836568    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhpdx\" (UniqueName: \"kubernetes.io/projected/a811bb1a-42b1-4184-b123-c295547116d2-kube-api-access-hhpdx\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:44.837453 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:44.837425    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-istio-envoy\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:45.124703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:45.123978    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:45.401504 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:45.400385    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:48:45.465766 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:45.462845    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:48:45.465766 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:45.463239    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:48:45.565858 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:48:45.564889    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:48:45.632704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:45.631766    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded"
Oct 31 00:48:45.956004 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:45.894939    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:48:46.507705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:46.502800    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz" podUID="b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:48:46.941080 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:46.941024    4327 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded"
Oct 31 00:48:47.154426 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:47.153418    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:48:47.234891 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:47.234727    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:48:47.236614 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:47.236578    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:48:47.236867 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:48:47.235979    4327 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.296s"
Oct 31 00:48:47.237616 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:47.237600    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:48:47.243162 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:47.243134    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"]
Oct 31 00:48:47.243522 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:47.236922    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:48:47.254569 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:47.254011    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:48:48.204619 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:48.204580    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh" event={"ID":"a811bb1a-42b1-4184-b123-c295547116d2","Type":"ContainerStarted","Data":"8d72e1c71501630b0c3c7adcfd48ba45fda3046ef9b961815a15a8b58b5adf13"}
Oct 31 00:48:48.205060 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:48.204647    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh" event={"ID":"a811bb1a-42b1-4184-b123-c295547116d2","Type":"ContainerStarted","Data":"16b5cabb7c6178ed9cbeffb17db136fd771cd5813336543f26cb5ce9c956f77c"}
Oct 31 00:48:48.264001 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:48.263948    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh" podStartSLOduration=5.263928276 podStartE2EDuration="5.263928276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:48.262312836 +0000 UTC m=+1788.484348377" watchObservedRunningTime="2026-10-31 00:48:48.263928276 +0000 UTC m=+1788.485963825"
Oct 31 00:48:49.124592 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:49.124551    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:49.130557 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:49.130528    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:49.213562 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:49.213534    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:49.214541 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:49.214520    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:48:52.950866 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:52.950829    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx"]
Oct 31 00:48:52.955584 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:52.955559    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx"
Oct 31 00:48:52.977338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:52.977305    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx"]
Oct 31 00:48:52.982893 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:52.982863    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-714f8f\""
Oct 31 00:48:52.984828 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:52.984798    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-714f8f-dockercfg-9zmdg\""
Oct 31 00:48:53.278766 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:53.115627    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpt59\" (UniqueName: \"kubernetes.io/projected/eab68ddb-030b-40ba-a1f9-bac24591237e-kube-api-access-wpt59\") pod \"payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx\" (UID: \"eab68ddb-030b-40ba-a1f9-bac24591237e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx"
Oct 31 00:48:53.279090 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:53.279050    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab68ddb-030b-40ba-a1f9-bac24591237e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx\" (UID: \"eab68ddb-030b-40ba-a1f9-bac24591237e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx"
Oct 31 00:48:53.498507 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:53.498468    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpt59\" (UniqueName: \"kubernetes.io/projected/eab68ddb-030b-40ba-a1f9-bac24591237e-kube-api-access-wpt59\") pod \"payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx\" (UID: \"eab68ddb-030b-40ba-a1f9-bac24591237e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx"
Oct 31 00:48:53.498838 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:53.498815    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab68ddb-030b-40ba-a1f9-bac24591237e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx\" (UID: \"eab68ddb-030b-40ba-a1f9-bac24591237e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx"
Oct 31 00:48:53.500629 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:53.500595    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab68ddb-030b-40ba-a1f9-bac24591237e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx\" (UID: \"eab68ddb-030b-40ba-a1f9-bac24591237e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx"
Oct 31 00:48:53.577225 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:53.574664    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpt59\" (UniqueName: \"kubernetes.io/projected/eab68ddb-030b-40ba-a1f9-bac24591237e-kube-api-access-wpt59\") pod \"payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx\" (UID: \"eab68ddb-030b-40ba-a1f9-bac24591237e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx"
Oct 31 00:48:53.870786 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:53.870755    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx"
Oct 31 00:48:54.184901 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:54.184873    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx"]
Oct 31 00:48:54.185415 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:48:54.185389    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeab68ddb_030b_40ba_a1f9_bac24591237e.slice/crio-c859b33227dea07397469719d490536b7d0d5a5fe95df21e7fc33ddb8248a09b WatchSource:0}: Error finding container c859b33227dea07397469719d490536b7d0d5a5fe95df21e7fc33ddb8248a09b: Status 404 returned error can't find the container with id c859b33227dea07397469719d490536b7d0d5a5fe95df21e7fc33ddb8248a09b
Oct 31 00:48:54.364287 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:54.364251    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx" event={"ID":"eab68ddb-030b-40ba-a1f9-bac24591237e","Type":"ContainerStarted","Data":"ab8a8b3f59b553f92097ccd7ebc677c631e39eefa02ce1c0e536f713408ed119"}
Oct 31 00:48:54.364510 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:54.364304    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx" event={"ID":"eab68ddb-030b-40ba-a1f9-bac24591237e","Type":"ContainerStarted","Data":"c859b33227dea07397469719d490536b7d0d5a5fe95df21e7fc33ddb8248a09b"}
Oct 31 00:48:54.364510 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:54.364341    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx"
Oct 31 00:48:54.439587 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:48:54.439474    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx" podStartSLOduration=2.439460136 podStartE2EDuration="2.439460136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:54.435865141 +0000 UTC m=+1794.657900684" watchObservedRunningTime="2026-10-31 00:48:54.439460136 +0000 UTC m=+1794.661495678"
Oct 31 00:49:05.376209 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:05.376177    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx"
Oct 31 00:49:15.424545 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:49:15.424446    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:49:19.646489 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:19.646450    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2"]
Oct 31 00:49:19.652567 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:19.652542    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2"
Oct 31 00:49:19.670632 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:19.670582    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-3b1945-dockercfg-fxtr8\""
Oct 31 00:49:19.671047 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:19.671023    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-3b1945\""
Oct 31 00:49:19.672889 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:19.672862    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2"]
Oct 31 00:49:19.836705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:19.827035    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5740efc6-1a2d-4762-93ad-ce5310f03c7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2\" (UID: \"5740efc6-1a2d-4762-93ad-ce5310f03c7a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2"
Oct 31 00:49:19.836705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:19.827168    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv6cb\" (UniqueName: \"kubernetes.io/projected/5740efc6-1a2d-4762-93ad-ce5310f03c7a-kube-api-access-zv6cb\") pod \"payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2\" (UID: \"5740efc6-1a2d-4762-93ad-ce5310f03c7a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2"
Oct 31 00:49:19.991771 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:19.980955    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5740efc6-1a2d-4762-93ad-ce5310f03c7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2\" (UID: \"5740efc6-1a2d-4762-93ad-ce5310f03c7a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2"
Oct 31 00:49:19.991771 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:19.981237    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv6cb\" (UniqueName: \"kubernetes.io/projected/5740efc6-1a2d-4762-93ad-ce5310f03c7a-kube-api-access-zv6cb\") pod \"payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2\" (UID: \"5740efc6-1a2d-4762-93ad-ce5310f03c7a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2"
Oct 31 00:49:19.991771 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:19.983157    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5740efc6-1a2d-4762-93ad-ce5310f03c7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2\" (UID: \"5740efc6-1a2d-4762-93ad-ce5310f03c7a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2"
Oct 31 00:49:20.176489 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:20.166659    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv6cb\" (UniqueName: \"kubernetes.io/projected/5740efc6-1a2d-4762-93ad-ce5310f03c7a-kube-api-access-zv6cb\") pod \"payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2\" (UID: \"5740efc6-1a2d-4762-93ad-ce5310f03c7a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2"
Oct 31 00:49:20.287310 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:20.277741    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2"
Oct 31 00:49:20.287310 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:20.283962    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2"]
Oct 31 00:49:20.326561 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:20.307901    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2"]
Oct 31 00:49:20.326561 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:20.308036    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2"
Oct 31 00:49:20.427957 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:20.427216    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg2s4\" (UniqueName: \"kubernetes.io/projected/d8424735-42e3-406b-897c-9abe18751264-kube-api-access-dg2s4\") pod \"payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2\" (UID: \"d8424735-42e3-406b-897c-9abe18751264\") " pod="openshift-ingress/payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2"
Oct 31 00:49:20.427957 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:20.427569    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8424735-42e3-406b-897c-9abe18751264-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2\" (UID: \"d8424735-42e3-406b-897c-9abe18751264\") " pod="openshift-ingress/payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2"
Oct 31 00:49:20.528863 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:20.528819    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg2s4\" (UniqueName: \"kubernetes.io/projected/d8424735-42e3-406b-897c-9abe18751264-kube-api-access-dg2s4\") pod \"payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2\" (UID: \"d8424735-42e3-406b-897c-9abe18751264\") " pod="openshift-ingress/payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2"
Oct 31 00:49:20.529165 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:20.529134    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8424735-42e3-406b-897c-9abe18751264-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2\" (UID: \"d8424735-42e3-406b-897c-9abe18751264\") " pod="openshift-ingress/payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2"
Oct 31 00:49:20.530096 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:20.530072    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8424735-42e3-406b-897c-9abe18751264-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2\" (UID: \"d8424735-42e3-406b-897c-9abe18751264\") " pod="openshift-ingress/payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2"
Oct 31 00:49:20.553803 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:20.553721    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg2s4\" (UniqueName: \"kubernetes.io/projected/d8424735-42e3-406b-897c-9abe18751264-kube-api-access-dg2s4\") pod \"payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2\" (UID: \"d8424735-42e3-406b-897c-9abe18751264\") " pod="openshift-ingress/payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2"
Oct 31 00:49:20.644828 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:49:20.644793    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5740efc6_1a2d_4762_93ad_ce5310f03c7a.slice/crio-191185dd924bc3df99307b75b8f2e3f15465909f898e1e0fc5344fd0043f1296 WatchSource:0}: Error finding container 191185dd924bc3df99307b75b8f2e3f15465909f898e1e0fc5344fd0043f1296: Status 404 returned error can't find the container with id 191185dd924bc3df99307b75b8f2e3f15465909f898e1e0fc5344fd0043f1296
Oct 31 00:49:20.646433 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:20.646401    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2"]
Oct 31 00:49:20.675708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:20.675664    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2"
Oct 31 00:49:21.088645 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:21.088601    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2"]
Oct 31 00:49:21.089776 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:49:21.089352    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8424735_42e3_406b_897c_9abe18751264.slice/crio-1e935a424da1ec5e02ad8a38c77e212dbc90b88f95298b3e787684f77c92284f WatchSource:0}: Error finding container 1e935a424da1ec5e02ad8a38c77e212dbc90b88f95298b3e787684f77c92284f: Status 404 returned error can't find the container with id 1e935a424da1ec5e02ad8a38c77e212dbc90b88f95298b3e787684f77c92284f
Oct 31 00:49:21.210966 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:21.210928    4327 generic.go:356] "Generic (PLEG): container finished" podID="aca0da87-ad0a-45cc-a022-d4946e7fab22" containerID="2997c24959fba21be98fced109a4b85993b2dfc6151f35d7bc4c237aa3180251" exitCode=0
Oct 31 00:49:21.211134 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:21.211119    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh" event={"ID":"aca0da87-ad0a-45cc-a022-d4946e7fab22","Type":"ContainerDied","Data":"2997c24959fba21be98fced109a4b85993b2dfc6151f35d7bc4c237aa3180251"}
Oct 31 00:49:21.211736 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:21.211656    4327 scope.go:122] "RemoveContainer" containerID="2997c24959fba21be98fced109a4b85993b2dfc6151f35d7bc4c237aa3180251"
Oct 31 00:49:21.214222 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:21.214178    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:49:21.214323 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:21.214281    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:49:21.214392 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:21.214329    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:49:21.216055 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:21.216028    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2" event={"ID":"5740efc6-1a2d-4762-93ad-ce5310f03c7a","Type":"ContainerStarted","Data":"ce23b983f8a1be07e3e118480789461c4bc345b4404284a7c7405c2b3609a105"}
Oct 31 00:49:21.216189 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:21.216178    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2"
Oct 31 00:49:21.216262 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:21.216253    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2" event={"ID":"5740efc6-1a2d-4762-93ad-ce5310f03c7a","Type":"ContainerStarted","Data":"191185dd924bc3df99307b75b8f2e3f15465909f898e1e0fc5344fd0043f1296"}
Oct 31 00:49:21.218034 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:21.218011    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2" event={"ID":"d8424735-42e3-406b-897c-9abe18751264","Type":"ContainerStarted","Data":"1e935a424da1ec5e02ad8a38c77e212dbc90b88f95298b3e787684f77c92284f"}
Oct 31 00:49:21.320125 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:21.320072    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2" podStartSLOduration=2.320055593 podStartE2EDuration="2.320055593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:21.31163896 +0000 UTC m=+1821.533674503" watchObservedRunningTime="2026-10-31 00:49:21.320055593 +0000 UTC m=+1821.542091139"
Oct 31 00:49:22.768704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:22.767735    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2" event={"ID":"d8424735-42e3-406b-897c-9abe18751264","Type":"ContainerStarted","Data":"b8df9e7bb2cf8cb0365b7fa7931a95863795472f1db50ad11532259f45311a93"}
Oct 31 00:49:22.772739 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:22.770205    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2"
Oct 31 00:49:22.785026 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:22.784987    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh" event={"ID":"aca0da87-ad0a-45cc-a022-d4946e7fab22","Type":"ContainerStarted","Data":"af2e4eca3107c1082dd2626643c999ae9eb38e7f41b782c99e52982e2e64ad3f"}
Oct 31 00:49:22.847875 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:22.847801    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2" podStartSLOduration=3.84778337 podStartE2EDuration="3.84778337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:22.846231766 +0000 UTC m=+1823.068267308" watchObservedRunningTime="2026-10-31 00:49:22.84778337 +0000 UTC m=+1823.069818912"
Oct 31 00:49:23.573705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:23.570599    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:49:23.573705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:23.570759    4327 patch_prober.go:29] interesting pod/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 31 00:49:23.573705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:23.570805    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh" podUID="aca0da87-ad0a-45cc-a022-d4946e7fab22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 31 00:49:24.565392 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:24.565354    4327 patch_prober.go:29] interesting pod/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 31 00:49:24.565947 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:24.565415    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh" podUID="aca0da87-ad0a-45cc-a022-d4946e7fab22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 31 00:49:25.583711 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:25.566033    4327 patch_prober.go:29] interesting pod/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 31 00:49:25.636437 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:25.636397    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh" podUID="aca0da87-ad0a-45cc-a022-d4946e7fab22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 31 00:49:26.565200 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:26.565165    4327 patch_prober.go:29] interesting pod/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 31 00:49:26.565394 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:26.565220    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh" podUID="aca0da87-ad0a-45cc-a022-d4946e7fab22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 31 00:49:27.565063 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:27.565023    4327 patch_prober.go:29] interesting pod/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 31 00:49:27.565576 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:27.565096    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh" podUID="aca0da87-ad0a-45cc-a022-d4946e7fab22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 31 00:49:27.591760 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:27.591724    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:49:28.568559 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:28.568103    4327 patch_prober.go:29] interesting pod/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 31 00:49:28.568559 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:28.568172    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh" podUID="aca0da87-ad0a-45cc-a022-d4946e7fab22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 31 00:49:29.582795 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:29.582765    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:49:29.583850 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:29.583827    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:49:31.548180 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:31.548145    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65"]
Oct 31 00:49:31.553378 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:31.553354    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65"
Oct 31 00:49:31.575420 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:31.575388    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5"]
Oct 31 00:49:31.583166 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:31.583136    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-3d21c7-dockercfg-bt9mm\""
Oct 31 00:49:31.583166 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:31.583137    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-3d21c7\""
Oct 31 00:49:31.592905 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:31.592875    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65"]
Oct 31 00:49:31.593218 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:31.593165    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5"
Oct 31 00:49:31.624238 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:31.624208    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5"]
Oct 31 00:49:31.693879 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:31.680041    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9751c672-027d-40c3-9da4-a18c4671f8fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5\" (UID: \"9751c672-027d-40c3-9da4-a18c4671f8fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5"
Oct 31 00:49:31.693879 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:31.680149    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39a92afd-325e-4e33-9d0c-6b9fde7a49c3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65\" (UID: \"39a92afd-325e-4e33-9d0c-6b9fde7a49c3\") " pod="openshift-ingress/payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65"
Oct 31 00:49:31.693879 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:31.680225    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cmjr\" (UniqueName: \"kubernetes.io/projected/9751c672-027d-40c3-9da4-a18c4671f8fa-kube-api-access-4cmjr\") pod \"payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5\" (UID: \"9751c672-027d-40c3-9da4-a18c4671f8fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5"
Oct 31 00:49:31.693879 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:31.680458    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsqdx\" (UniqueName: \"kubernetes.io/projected/39a92afd-325e-4e33-9d0c-6b9fde7a49c3-kube-api-access-gsqdx\") pod \"payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65\" (UID: \"39a92afd-325e-4e33-9d0c-6b9fde7a49c3\") " pod="openshift-ingress/payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65"
Oct 31 00:49:31.788660 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:31.788608    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsqdx\" (UniqueName: \"kubernetes.io/projected/39a92afd-325e-4e33-9d0c-6b9fde7a49c3-kube-api-access-gsqdx\") pod \"payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65\" (UID: \"39a92afd-325e-4e33-9d0c-6b9fde7a49c3\") " pod="openshift-ingress/payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65"
Oct 31 00:49:31.789126 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:31.789101    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9751c672-027d-40c3-9da4-a18c4671f8fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5\" (UID: \"9751c672-027d-40c3-9da4-a18c4671f8fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5"
Oct 31 00:49:31.806759 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:31.789294    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39a92afd-325e-4e33-9d0c-6b9fde7a49c3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65\" (UID: \"39a92afd-325e-4e33-9d0c-6b9fde7a49c3\") " pod="openshift-ingress/payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65"
Oct 31 00:49:31.807115 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:31.807076    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cmjr\" (UniqueName: \"kubernetes.io/projected/9751c672-027d-40c3-9da4-a18c4671f8fa-kube-api-access-4cmjr\") pod \"payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5\" (UID: \"9751c672-027d-40c3-9da4-a18c4671f8fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5"
Oct 31 00:49:31.852706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:31.810334    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39a92afd-325e-4e33-9d0c-6b9fde7a49c3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65\" (UID: \"39a92afd-325e-4e33-9d0c-6b9fde7a49c3\") " pod="openshift-ingress/payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65"
Oct 31 00:49:32.096917 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:31.919613    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9751c672-027d-40c3-9da4-a18c4671f8fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5\" (UID: \"9751c672-027d-40c3-9da4-a18c4671f8fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5"
Oct 31 00:49:32.097493 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:31.979159    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cmjr\" (UniqueName: \"kubernetes.io/projected/9751c672-027d-40c3-9da4-a18c4671f8fa-kube-api-access-4cmjr\") pod \"payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5\" (UID: \"9751c672-027d-40c3-9da4-a18c4671f8fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5"
Oct 31 00:49:32.097646 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:32.027169    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsqdx\" (UniqueName: \"kubernetes.io/projected/39a92afd-325e-4e33-9d0c-6b9fde7a49c3-kube-api-access-gsqdx\") pod \"payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65\" (UID: \"39a92afd-325e-4e33-9d0c-6b9fde7a49c3\") " pod="openshift-ingress/payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65"
Oct 31 00:49:32.179106 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:32.179013    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65"
Oct 31 00:49:32.213202 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:32.213154    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5"
Oct 31 00:49:32.788250 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:32.788223    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2"
Oct 31 00:49:33.100644 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:49:33.100492    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a92afd_325e_4e33_9d0c_6b9fde7a49c3.slice/crio-df4a71c3e7079d3f01f02468a25e636954a9900f25911680acf03ca36b1e8cb8 WatchSource:0}: Error finding container df4a71c3e7079d3f01f02468a25e636954a9900f25911680acf03ca36b1e8cb8: Status 404 returned error can't find the container with id df4a71c3e7079d3f01f02468a25e636954a9900f25911680acf03ca36b1e8cb8
Oct 31 00:49:33.150613 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:33.150575    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65"]
Oct 31 00:49:33.253508 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:33.253457    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5"]
Oct 31 00:49:33.893029 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:33.892993    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2"
Oct 31 00:49:34.264855 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:34.264810    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5" event={"ID":"9751c672-027d-40c3-9da4-a18c4671f8fa","Type":"ContainerStarted","Data":"386f2360276b42100359c93542d91148ae8db81181178a25a90d6302b8d09849"}
Oct 31 00:49:34.265060 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:34.264870    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5" event={"ID":"9751c672-027d-40c3-9da4-a18c4671f8fa","Type":"ContainerStarted","Data":"3aae15d180e71aa3c0387f23aa9cfdfc4cf7c2981a20d37943c07348110cf02d"}
Oct 31 00:49:34.265060 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:34.264915    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5"
Oct 31 00:49:34.266800 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:34.266774    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65" event={"ID":"39a92afd-325e-4e33-9d0c-6b9fde7a49c3","Type":"ContainerStarted","Data":"8eba09a60b769dd64a38906d0984877a8df27a1ff98201103a3b55b2309b8a9b"}
Oct 31 00:49:34.266937 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:34.266815    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65" event={"ID":"39a92afd-325e-4e33-9d0c-6b9fde7a49c3","Type":"ContainerStarted","Data":"df4a71c3e7079d3f01f02468a25e636954a9900f25911680acf03ca36b1e8cb8"}
Oct 31 00:49:34.266937 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:34.266846    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65"
Oct 31 00:49:34.315922 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:34.315864    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5" podStartSLOduration=3.315849294 podStartE2EDuration="3.315849294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:34.310283816 +0000 UTC m=+1834.532319369" watchObservedRunningTime="2026-10-31 00:49:34.315849294 +0000 UTC m=+1834.537884839"
Oct 31 00:49:34.356694 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:34.356623    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65" podStartSLOduration=3.35660657 podStartE2EDuration="3.35660657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:34.35103568 +0000 UTC m=+1834.573071221" watchObservedRunningTime="2026-10-31 00:49:34.35660657 +0000 UTC m=+1834.578642111"
Oct 31 00:49:45.278435 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:45.278404    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65"
Oct 31 00:49:45.278888 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:45.278491    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5"
Oct 31 00:49:45.294785 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:49:45.294750    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:49:48.767977 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:48.767944    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"]
Oct 31 00:49:48.768385 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:48.768157    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8" podUID="c9c74e25-811d-414f-b093-a907ca85631d" containerName="istio-proxy" containerID="cri-o://925e9b469f952663dde1cd4de03c94b30b5bb689422e8a82e005a6d2723da3ab" gracePeriod=30
Oct 31 00:49:48.784693 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:48.782369    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"]
Oct 31 00:49:48.784693 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:48.782618    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj" podUID="148ce8dc-1145-488d-940e-111d5cb3d9fc" containerName="istio-proxy" containerID="cri-o://56bfc919e76e2fe0ec2eb4387ebc5f72fc19cce0225ae22f678fb3c7ad97f710" gracePeriod=30
Oct 31 00:49:48.802302 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:48.802260    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"]
Oct 31 00:49:48.805027 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:48.804989    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58" podUID="ed99fa70-689b-472e-8e25-62e70c3f2972" containerName="istio-proxy" containerID="cri-o://46e9236e088d6783f24a0109d50cf8c0c2c4a666f91d35db389ac0e7db70f1ff" gracePeriod=30
Oct 31 00:49:49.742627 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:49.742593    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"]
Oct 31 00:49:49.742852 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:49.742829    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr" podUID="dc70d4a7-43f4-4a03-883b-2035e3f2c137" containerName="istio-proxy" containerID="cri-o://afdc6044c6f2475b0a04c55a3da19a2cef18cc858d5c6902f9ec24bfbd3d6b6b" gracePeriod=30
Oct 31 00:49:49.756422 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:49.756390    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"]
Oct 31 00:49:49.756642 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:49.756610    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x" podUID="0260aaaa-f469-439d-91a1-45e0c4aab165" containerName="istio-proxy" containerID="cri-o://93d27ffdf13e5fa3cad8c5a67f1f02e8a9a5808c0bfecc5e7b8f829b5f3a3782" gracePeriod=30
Oct 31 00:49:49.762085 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:49.762053    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"]
Oct 31 00:49:49.762320 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:49.762298    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj" podUID="47d509de-a07d-49ba-9a35-67873b87041b" containerName="istio-proxy" containerID="cri-o://d783c099b11ddd7cc224d3fa18bc03971543d150ed3069f48f24a36d729b8b2a" gracePeriod=30
Oct 31 00:49:49.766891 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:49.766867    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"]
Oct 31 00:49:49.767119 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:49.767095    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97" podUID="be256a75-3d28-43d7-9b5d-56bf5498f0d1" containerName="istio-proxy" containerID="cri-o://8ae1dd10b9a8f5cb027e10bf10826d754e8bc9dd6bff45b64e005a769c8aa58f" gracePeriod=30
Oct 31 00:49:49.937721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:49.937663    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"]
Oct 31 00:49:49.938324 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:49.937958    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd" podUID="49a4ff5e-3ec9-45ce-88c2-a9b6308a821e" containerName="istio-proxy" containerID="cri-o://6ce9aee27c6c3412514e9a07f4333228e5cb3a960b110d3a1cc34ce89476d265" gracePeriod=30
Oct 31 00:49:49.946148 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:49.946121    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"]
Oct 31 00:49:49.946346 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:49.946319    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch" podUID="80939755-d4c4-4c9e-82dc-6f4349c25415" containerName="istio-proxy" containerID="cri-o://b0c92f0638452a6b2c72c15d0991950d63dbc7fd10ea4e7fe0fc6173ec8178eb" gracePeriod=30
Oct 31 00:49:49.952099 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:49.952071    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"]
Oct 31 00:49:49.952339 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:49.952312    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx" podUID="ab534e5f-cfc2-4be6-acf5-bc00c9fa957f" containerName="istio-proxy" containerID="cri-o://48274dd203ecd370073dda9e8f7e96b8cf8d68fc71f6bc83e57a8e3cef537205" gracePeriod=30
Oct 31 00:49:49.959050 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:49.958965    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"]
Oct 31 00:49:49.959363 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:49.959269    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2" podUID="ce1ceaa6-3845-4dc6-8455-974804695d4c" containerName="istio-proxy" containerID="cri-o://b9062b534627de3b117cabddc170a59309e3cd8e61152f5b6e47834a3e6cb9da" gracePeriod=30
Oct 31 00:49:52.620623 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:52.620578    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:49:52.621220 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:52.620660    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x" podUID="0260aaaa-f469-439d-91a1-45e0c4aab165" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:49:52.780851 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:52.780812    4327 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:49:52.781143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:52.781107    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj" podUID="47d509de-a07d-49ba-9a35-67873b87041b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:49:54.516638 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.516608    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:49:54.593127 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.592067    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-workload-certs\") pod \"c9c74e25-811d-414f-b093-a907ca85631d\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") "
Oct 31 00:49:54.593127 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.592177    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-istio-data\") pod \"c9c74e25-811d-414f-b093-a907ca85631d\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") "
Oct 31 00:49:54.593127 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.592228    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-workload-socket\") pod \"c9c74e25-811d-414f-b093-a907ca85631d\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") "
Oct 31 00:49:54.593127 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.592278    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c74e25-811d-414f-b093-a907ca85631d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c74e25-811d-414f-b093-a907ca85631d-istiod-ca-cert\") pod \"c9c74e25-811d-414f-b093-a907ca85631d\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") "
Oct 31 00:49:54.593127 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.592352    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-credential-socket\") pod \"c9c74e25-811d-414f-b093-a907ca85631d\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") "
Oct 31 00:49:54.593127 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.592379    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c74e25-811d-414f-b093-a907ca85631d-istio-token\" (UniqueName: \"kubernetes.io/projected/c9c74e25-811d-414f-b093-a907ca85631d-istio-token\") pod \"c9c74e25-811d-414f-b093-a907ca85631d\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") "
Oct 31 00:49:54.593127 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.592439    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c74e25-811d-414f-b093-a907ca85631d-kube-api-access-dlxkt\" (UniqueName: \"kubernetes.io/projected/c9c74e25-811d-414f-b093-a907ca85631d-kube-api-access-dlxkt\") pod \"c9c74e25-811d-414f-b093-a907ca85631d\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") "
Oct 31 00:49:54.593127 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.592584    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-istio-envoy\") pod \"c9c74e25-811d-414f-b093-a907ca85631d\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") "
Oct 31 00:49:54.593127 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.592697    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9c74e25-811d-414f-b093-a907ca85631d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c74e25-811d-414f-b093-a907ca85631d-istio-podinfo\") pod \"c9c74e25-811d-414f-b093-a907ca85631d\" (UID: \"c9c74e25-811d-414f-b093-a907ca85631d\") "
Oct 31 00:49:54.593127 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.592734    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-workload-socket" pod "c9c74e25-811d-414f-b093-a907ca85631d" (UID: "c9c74e25-811d-414f-b093-a907ca85631d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:54.593127 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.592755    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-workload-certs" pod "c9c74e25-811d-414f-b093-a907ca85631d" (UID: "c9c74e25-811d-414f-b093-a907ca85631d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:54.593127 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.593056    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-credential-socket" pod "c9c74e25-811d-414f-b093-a907ca85631d" (UID: "c9c74e25-811d-414f-b093-a907ca85631d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:54.593923 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.593627    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:54.593923 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.593652    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:54.593923 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.593667    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:54.593923 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.593740    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-istio-data" pod "c9c74e25-811d-414f-b093-a907ca85631d" (UID: "c9c74e25-811d-414f-b093-a907ca85631d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:54.594255 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.594200    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c74e25-811d-414f-b093-a907ca85631d-istiod-ca-cert" pod "c9c74e25-811d-414f-b093-a907ca85631d" (UID: "c9c74e25-811d-414f-b093-a907ca85631d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:49:54.597260 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.597231    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-istio-envoy" pod "c9c74e25-811d-414f-b093-a907ca85631d" (UID: "c9c74e25-811d-414f-b093-a907ca85631d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:54.597732 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.597665    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9c74e25-811d-414f-b093-a907ca85631d-istio-podinfo" pod "c9c74e25-811d-414f-b093-a907ca85631d" (UID: "c9c74e25-811d-414f-b093-a907ca85631d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:49:54.597833 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.597764    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c74e25-811d-414f-b093-a907ca85631d-kube-api-access-dlxkt" pod "c9c74e25-811d-414f-b093-a907ca85631d" (UID: "c9c74e25-811d-414f-b093-a907ca85631d"). InnerVolumeSpecName "kube-api-access-dlxkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:54.599433 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.599406    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c74e25-811d-414f-b093-a907ca85631d-istio-token" pod "c9c74e25-811d-414f-b093-a907ca85631d" (UID: "c9c74e25-811d-414f-b093-a907ca85631d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:54.650148 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.648214    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:49:54.676007 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.675973    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:49:54.683288 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.683255    4327 generic.go:356] "Generic (PLEG): container finished" podID="c9c74e25-811d-414f-b093-a907ca85631d" containerID="925e9b469f952663dde1cd4de03c94b30b5bb689422e8a82e005a6d2723da3ab" exitCode=0
Oct 31 00:49:54.683440 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.683296    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"
Oct 31 00:49:54.683440 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.683332    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8" event={"ID":"c9c74e25-811d-414f-b093-a907ca85631d","Type":"ContainerDied","Data":"925e9b469f952663dde1cd4de03c94b30b5bb689422e8a82e005a6d2723da3ab"}
Oct 31 00:49:54.683440 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.683390    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8" event={"ID":"c9c74e25-811d-414f-b093-a907ca85631d","Type":"ContainerDied","Data":"b437fc9315590ad3f22ffc6e9681a14dc12d326119ff3b28ff6cfbf58a3ef52f"}
Oct 31 00:49:54.683440 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.683419    4327 scope.go:122] "RemoveContainer" containerID="925e9b469f952663dde1cd4de03c94b30b5bb689422e8a82e005a6d2723da3ab"
Oct 31 00:49:54.685083 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.685059    4327 generic.go:356] "Generic (PLEG): container finished" podID="148ce8dc-1145-488d-940e-111d5cb3d9fc" containerID="56bfc919e76e2fe0ec2eb4387ebc5f72fc19cce0225ae22f678fb3c7ad97f710" exitCode=0
Oct 31 00:49:54.685215 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.685114    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"
Oct 31 00:49:54.685215 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.685192    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj" event={"ID":"148ce8dc-1145-488d-940e-111d5cb3d9fc","Type":"ContainerDied","Data":"56bfc919e76e2fe0ec2eb4387ebc5f72fc19cce0225ae22f678fb3c7ad97f710"}
Oct 31 00:49:54.685317 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.685235    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj" event={"ID":"148ce8dc-1145-488d-940e-111d5cb3d9fc","Type":"ContainerDied","Data":"34380236d989edbd1abba0ac7eaf05175fd072a6df678f9294da40787f3ff53f"}
Oct 31 00:49:54.686834 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.686808    4327 generic.go:356] "Generic (PLEG): container finished" podID="ed99fa70-689b-472e-8e25-62e70c3f2972" containerID="46e9236e088d6783f24a0109d50cf8c0c2c4a666f91d35db389ac0e7db70f1ff" exitCode=0
Oct 31 00:49:54.686942 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.686841    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58" event={"ID":"ed99fa70-689b-472e-8e25-62e70c3f2972","Type":"ContainerDied","Data":"46e9236e088d6783f24a0109d50cf8c0c2c4a666f91d35db389ac0e7db70f1ff"}
Oct 31 00:49:54.686942 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.686861    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"
Oct 31 00:49:54.686942 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.686897    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58" event={"ID":"ed99fa70-689b-472e-8e25-62e70c3f2972","Type":"ContainerDied","Data":"b405b0ac22fc86dcf476df2907768675da3e5f82e710db143551a3a9c6e12efb"}
Oct 31 00:49:54.699244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.695799    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-podinfo\") pod \"148ce8dc-1145-488d-940e-111d5cb3d9fc\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") "
Oct 31 00:49:54.699244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.695918    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-data\") pod \"148ce8dc-1145-488d-940e-111d5cb3d9fc\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") "
Oct 31 00:49:54.699244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.695975    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-credential-socket\") pod \"148ce8dc-1145-488d-940e-111d5cb3d9fc\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") "
Oct 31 00:49:54.699244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.696028    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/148ce8dc-1145-488d-940e-111d5cb3d9fc-kube-api-access-56cqt\" (UniqueName: \"kubernetes.io/projected/148ce8dc-1145-488d-940e-111d5cb3d9fc-kube-api-access-56cqt\") pod \"148ce8dc-1145-488d-940e-111d5cb3d9fc\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") "
Oct 31 00:49:54.699244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.696055    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-token\" (UniqueName: \"kubernetes.io/projected/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-token\") pod \"ed99fa70-689b-472e-8e25-62e70c3f2972\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") "
Oct 31 00:49:54.699244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.696107    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-workload-socket\") pod \"148ce8dc-1145-488d-940e-111d5cb3d9fc\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") "
Oct 31 00:49:54.699244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.696140    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed99fa70-689b-472e-8e25-62e70c3f2972-kube-api-access-pq2z2\" (UniqueName: \"kubernetes.io/projected/ed99fa70-689b-472e-8e25-62e70c3f2972-kube-api-access-pq2z2\") pod \"ed99fa70-689b-472e-8e25-62e70c3f2972\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") "
Oct 31 00:49:54.699244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.696194    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/148ce8dc-1145-488d-940e-111d5cb3d9fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/148ce8dc-1145-488d-940e-111d5cb3d9fc-istiod-ca-cert\") pod \"148ce8dc-1145-488d-940e-111d5cb3d9fc\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") "
Oct 31 00:49:54.699244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.696255    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-podinfo\") pod \"ed99fa70-689b-472e-8e25-62e70c3f2972\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") "
Oct 31 00:49:54.699244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.696284    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-token\" (UniqueName: \"kubernetes.io/projected/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-token\") pod \"148ce8dc-1145-488d-940e-111d5cb3d9fc\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") "
Oct 31 00:49:54.699244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.696441    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed99fa70-689b-472e-8e25-62e70c3f2972-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed99fa70-689b-472e-8e25-62e70c3f2972-istiod-ca-cert\") pod \"ed99fa70-689b-472e-8e25-62e70c3f2972\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") "
Oct 31 00:49:54.699244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.696495    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-workload-certs\") pod \"ed99fa70-689b-472e-8e25-62e70c3f2972\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") "
Oct 31 00:49:54.699244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.696590    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-workload-certs\") pod \"148ce8dc-1145-488d-940e-111d5cb3d9fc\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") "
Oct 31 00:49:54.699244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.696691    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-envoy\") pod \"148ce8dc-1145-488d-940e-111d5cb3d9fc\" (UID: \"148ce8dc-1145-488d-940e-111d5cb3d9fc\") "
Oct 31 00:49:54.699244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.696755    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-credential-socket\") pod \"ed99fa70-689b-472e-8e25-62e70c3f2972\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") "
Oct 31 00:49:54.699244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.696850    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-data\") pod \"ed99fa70-689b-472e-8e25-62e70c3f2972\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") "
Oct 31 00:49:54.699244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.696940    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-envoy\") pod \"ed99fa70-689b-472e-8e25-62e70c3f2972\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") "
Oct 31 00:49:54.699244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.696986    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-workload-socket\") pod \"ed99fa70-689b-472e-8e25-62e70c3f2972\" (UID: \"ed99fa70-689b-472e-8e25-62e70c3f2972\") "
Oct 31 00:49:54.699244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.698587    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:54.699244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.698612    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c74e25-811d-414f-b093-a907ca85631d-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:54.699244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.698626    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c74e25-811d-414f-b093-a907ca85631d-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:54.699244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.698642    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c74e25-811d-414f-b093-a907ca85631d-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:54.699244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.698655    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c74e25-811d-414f-b093-a907ca85631d-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:54.699244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.698668    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlxkt\" (UniqueName: \"kubernetes.io/projected/c9c74e25-811d-414f-b093-a907ca85631d-kube-api-access-dlxkt\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:54.699244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.698918    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-workload-socket" pod "ed99fa70-689b-472e-8e25-62e70c3f2972" (UID: "ed99fa70-689b-472e-8e25-62e70c3f2972"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:54.701112 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.701081    4327 scope.go:122] "RemoveContainer" containerID="925e9b469f952663dde1cd4de03c94b30b5bb689422e8a82e005a6d2723da3ab"
Oct 31 00:49:54.701861 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:49:54.701538    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"925e9b469f952663dde1cd4de03c94b30b5bb689422e8a82e005a6d2723da3ab\": container with ID starting with 925e9b469f952663dde1cd4de03c94b30b5bb689422e8a82e005a6d2723da3ab not found: ID does not exist" containerID="925e9b469f952663dde1cd4de03c94b30b5bb689422e8a82e005a6d2723da3ab"
Oct 31 00:49:54.701861 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.701578    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"925e9b469f952663dde1cd4de03c94b30b5bb689422e8a82e005a6d2723da3ab"} err="failed to get container status \"925e9b469f952663dde1cd4de03c94b30b5bb689422e8a82e005a6d2723da3ab\": rpc error: code = NotFound desc = could not find container \"925e9b469f952663dde1cd4de03c94b30b5bb689422e8a82e005a6d2723da3ab\": container with ID starting with 925e9b469f952663dde1cd4de03c94b30b5bb689422e8a82e005a6d2723da3ab not found: ID does not exist"
Oct 31 00:49:54.701861 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.701602    4327 scope.go:122] "RemoveContainer" containerID="56bfc919e76e2fe0ec2eb4387ebc5f72fc19cce0225ae22f678fb3c7ad97f710"
Oct 31 00:49:54.701861 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.701804    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-podinfo" pod "148ce8dc-1145-488d-940e-111d5cb3d9fc" (UID: "148ce8dc-1145-488d-940e-111d5cb3d9fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:49:54.720643 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.720230    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-podinfo" pod "ed99fa70-689b-472e-8e25-62e70c3f2972" (UID: "ed99fa70-689b-472e-8e25-62e70c3f2972"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:49:54.720643 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.720510    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-workload-socket" pod "148ce8dc-1145-488d-940e-111d5cb3d9fc" (UID: "148ce8dc-1145-488d-940e-111d5cb3d9fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:54.741754 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.722244    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-data" pod "148ce8dc-1145-488d-940e-111d5cb3d9fc" (UID: "148ce8dc-1145-488d-940e-111d5cb3d9fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:54.741754 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.722887    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-data" pod "ed99fa70-689b-472e-8e25-62e70c3f2972" (UID: "ed99fa70-689b-472e-8e25-62e70c3f2972"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:54.741754 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.723138    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-workload-certs" pod "ed99fa70-689b-472e-8e25-62e70c3f2972" (UID: "ed99fa70-689b-472e-8e25-62e70c3f2972"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:54.741754 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.723422    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed99fa70-689b-472e-8e25-62e70c3f2972-istiod-ca-cert" pod "ed99fa70-689b-472e-8e25-62e70c3f2972" (UID: "ed99fa70-689b-472e-8e25-62e70c3f2972"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:49:54.741754 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.723656    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-workload-certs" pod "148ce8dc-1145-488d-940e-111d5cb3d9fc" (UID: "148ce8dc-1145-488d-940e-111d5cb3d9fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:54.741754 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.723716    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-credential-socket" pod "148ce8dc-1145-488d-940e-111d5cb3d9fc" (UID: "148ce8dc-1145-488d-940e-111d5cb3d9fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:54.741754 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.724161    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/148ce8dc-1145-488d-940e-111d5cb3d9fc-istiod-ca-cert" pod "148ce8dc-1145-488d-940e-111d5cb3d9fc" (UID: "148ce8dc-1145-488d-940e-111d5cb3d9fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:49:54.741754 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.724287    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-token" pod "148ce8dc-1145-488d-940e-111d5cb3d9fc" (UID: "148ce8dc-1145-488d-940e-111d5cb3d9fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:54.762866 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.762795    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-credential-socket" pod "ed99fa70-689b-472e-8e25-62e70c3f2972" (UID: "ed99fa70-689b-472e-8e25-62e70c3f2972"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:54.770881 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.770831    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-envoy" pod "148ce8dc-1145-488d-940e-111d5cb3d9fc" (UID: "148ce8dc-1145-488d-940e-111d5cb3d9fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:54.772338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.772285    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/148ce8dc-1145-488d-940e-111d5cb3d9fc-kube-api-access-56cqt" pod "148ce8dc-1145-488d-940e-111d5cb3d9fc" (UID: "148ce8dc-1145-488d-940e-111d5cb3d9fc"). InnerVolumeSpecName "kube-api-access-56cqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:54.773339 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.773289    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-envoy" pod "ed99fa70-689b-472e-8e25-62e70c3f2972" (UID: "ed99fa70-689b-472e-8e25-62e70c3f2972"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:54.776197 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.774014    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed99fa70-689b-472e-8e25-62e70c3f2972-kube-api-access-pq2z2" pod "ed99fa70-689b-472e-8e25-62e70c3f2972" (UID: "ed99fa70-689b-472e-8e25-62e70c3f2972"). InnerVolumeSpecName "kube-api-access-pq2z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:54.778481 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.778457    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"]
Oct 31 00:49:54.783133 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.780946    4327 scope.go:122] "RemoveContainer" containerID="56bfc919e76e2fe0ec2eb4387ebc5f72fc19cce0225ae22f678fb3c7ad97f710"
Oct 31 00:49:54.783133 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:49:54.781351    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56bfc919e76e2fe0ec2eb4387ebc5f72fc19cce0225ae22f678fb3c7ad97f710\": container with ID starting with 56bfc919e76e2fe0ec2eb4387ebc5f72fc19cce0225ae22f678fb3c7ad97f710 not found: ID does not exist" containerID="56bfc919e76e2fe0ec2eb4387ebc5f72fc19cce0225ae22f678fb3c7ad97f710"
Oct 31 00:49:54.783133 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.781388    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56bfc919e76e2fe0ec2eb4387ebc5f72fc19cce0225ae22f678fb3c7ad97f710"} err="failed to get container status \"56bfc919e76e2fe0ec2eb4387ebc5f72fc19cce0225ae22f678fb3c7ad97f710\": rpc error: code = NotFound desc = could not find container \"56bfc919e76e2fe0ec2eb4387ebc5f72fc19cce0225ae22f678fb3c7ad97f710\": container with ID starting with 56bfc919e76e2fe0ec2eb4387ebc5f72fc19cce0225ae22f678fb3c7ad97f710 not found: ID does not exist"
Oct 31 00:49:54.783133 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.781413    4327 scope.go:122] "RemoveContainer" containerID="46e9236e088d6783f24a0109d50cf8c0c2c4a666f91d35db389ac0e7db70f1ff"
Oct 31 00:49:54.792322 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.791100    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-nwkl8"]
Oct 31 00:49:54.795204 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.795171    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-token" pod "ed99fa70-689b-472e-8e25-62e70c3f2972" (UID: "ed99fa70-689b-472e-8e25-62e70c3f2972"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:54.800701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.800609    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:54.800701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.800636    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:54.800701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.800651    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:54.800701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.800662    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:54.800988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.800720    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:54.800988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.800737    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:54.800988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.800750    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:54.800988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.800766    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:54.800988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.800779    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56cqt\" (UniqueName: \"kubernetes.io/projected/148ce8dc-1145-488d-940e-111d5cb3d9fc-kube-api-access-56cqt\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:54.800988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.800792    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:54.800988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.800807    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:54.800988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.800822    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq2z2\" (UniqueName: \"kubernetes.io/projected/ed99fa70-689b-472e-8e25-62e70c3f2972-kube-api-access-pq2z2\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:54.800988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.800841    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/148ce8dc-1145-488d-940e-111d5cb3d9fc-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:54.800988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.800854    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed99fa70-689b-472e-8e25-62e70c3f2972-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:54.800988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.800867    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/148ce8dc-1145-488d-940e-111d5cb3d9fc-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:54.800988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.800880    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed99fa70-689b-472e-8e25-62e70c3f2972-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:54.800988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.800894    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed99fa70-689b-472e-8e25-62e70c3f2972-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:54.800988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:54.800906    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/148ce8dc-1145-488d-940e-111d5cb3d9fc-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.310643 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.310528    4327 scope.go:122] "RemoveContainer" containerID="46e9236e088d6783f24a0109d50cf8c0c2c4a666f91d35db389ac0e7db70f1ff"
Oct 31 00:49:55.311152 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:49:55.311115    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46e9236e088d6783f24a0109d50cf8c0c2c4a666f91d35db389ac0e7db70f1ff\": container with ID starting with 46e9236e088d6783f24a0109d50cf8c0c2c4a666f91d35db389ac0e7db70f1ff not found: ID does not exist" containerID="46e9236e088d6783f24a0109d50cf8c0c2c4a666f91d35db389ac0e7db70f1ff"
Oct 31 00:49:55.311275 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.311163    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46e9236e088d6783f24a0109d50cf8c0c2c4a666f91d35db389ac0e7db70f1ff"} err="failed to get container status \"46e9236e088d6783f24a0109d50cf8c0c2c4a666f91d35db389ac0e7db70f1ff\": rpc error: code = NotFound desc = could not find container \"46e9236e088d6783f24a0109d50cf8c0c2c4a666f91d35db389ac0e7db70f1ff\": container with ID starting with 46e9236e088d6783f24a0109d50cf8c0c2c4a666f91d35db389ac0e7db70f1ff not found: ID does not exist"
Oct 31 00:49:55.313485 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.313352    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:49:55.379059 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.379023    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"]
Oct 31 00:49:55.385973 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.385943    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:49:55.389474 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.389448    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-credential-socket\") pod \"47d509de-a07d-49ba-9a35-67873b87041b\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") "
Oct 31 00:49:55.389589 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.389514    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d509de-a07d-49ba-9a35-67873b87041b-kube-api-access-wq8sr\" (UniqueName: \"kubernetes.io/projected/47d509de-a07d-49ba-9a35-67873b87041b-kube-api-access-wq8sr\") pod \"47d509de-a07d-49ba-9a35-67873b87041b\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") "
Oct 31 00:49:55.389783 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.389630    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d509de-a07d-49ba-9a35-67873b87041b-istio-token\" (UniqueName: \"kubernetes.io/projected/47d509de-a07d-49ba-9a35-67873b87041b-istio-token\") pod \"47d509de-a07d-49ba-9a35-67873b87041b\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") "
Oct 31 00:49:55.389783 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.389707    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-istio-envoy\") pod \"47d509de-a07d-49ba-9a35-67873b87041b\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") "
Oct 31 00:49:55.389783 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.389762    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-workload-certs\") pod \"47d509de-a07d-49ba-9a35-67873b87041b\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") "
Oct 31 00:49:55.389994 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.389979    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-workload-socket\") pod \"47d509de-a07d-49ba-9a35-67873b87041b\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") "
Oct 31 00:49:55.390085 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.390072    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-istio-data\") pod \"47d509de-a07d-49ba-9a35-67873b87041b\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") "
Oct 31 00:49:55.390185 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.390173    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47d509de-a07d-49ba-9a35-67873b87041b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d509de-a07d-49ba-9a35-67873b87041b-istiod-ca-cert\") pod \"47d509de-a07d-49ba-9a35-67873b87041b\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") "
Oct 31 00:49:55.390237 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.390207    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47d509de-a07d-49ba-9a35-67873b87041b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d509de-a07d-49ba-9a35-67873b87041b-istio-podinfo\") pod \"47d509de-a07d-49ba-9a35-67873b87041b\" (UID: \"47d509de-a07d-49ba-9a35-67873b87041b\") "
Oct 31 00:49:55.390479 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.390456    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-ssq58"]
Oct 31 00:49:55.391338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.391308    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-credential-socket" pod "47d509de-a07d-49ba-9a35-67873b87041b" (UID: "47d509de-a07d-49ba-9a35-67873b87041b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.391532 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.391507    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-workload-socket" pod "47d509de-a07d-49ba-9a35-67873b87041b" (UID: "47d509de-a07d-49ba-9a35-67873b87041b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.391798 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.391761    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-istio-data" pod "47d509de-a07d-49ba-9a35-67873b87041b" (UID: "47d509de-a07d-49ba-9a35-67873b87041b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.391914 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.391796    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d509de-a07d-49ba-9a35-67873b87041b-istiod-ca-cert" pod "47d509de-a07d-49ba-9a35-67873b87041b" (UID: "47d509de-a07d-49ba-9a35-67873b87041b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:49:55.391914 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.391877    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-workload-certs" pod "47d509de-a07d-49ba-9a35-67873b87041b" (UID: "47d509de-a07d-49ba-9a35-67873b87041b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.393434 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.393404    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-istio-envoy" pod "47d509de-a07d-49ba-9a35-67873b87041b" (UID: "47d509de-a07d-49ba-9a35-67873b87041b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.394522 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.394478    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d509de-a07d-49ba-9a35-67873b87041b-kube-api-access-wq8sr" pod "47d509de-a07d-49ba-9a35-67873b87041b" (UID: "47d509de-a07d-49ba-9a35-67873b87041b"). InnerVolumeSpecName "kube-api-access-wq8sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:55.395388 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.395352    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d509de-a07d-49ba-9a35-67873b87041b-istio-token" pod "47d509de-a07d-49ba-9a35-67873b87041b" (UID: "47d509de-a07d-49ba-9a35-67873b87041b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:55.396314 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.396266    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47d509de-a07d-49ba-9a35-67873b87041b-istio-podinfo" pod "47d509de-a07d-49ba-9a35-67873b87041b" (UID: "47d509de-a07d-49ba-9a35-67873b87041b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:49:55.491342 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.491313    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-token\" (UniqueName: \"kubernetes.io/projected/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-token\") pod \"0260aaaa-f469-439d-91a1-45e0c4aab165\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") "
Oct 31 00:49:55.491498 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.491417    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-credential-socket\") pod \"0260aaaa-f469-439d-91a1-45e0c4aab165\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") "
Oct 31 00:49:55.491498 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.491492    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0260aaaa-f469-439d-91a1-45e0c4aab165-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0260aaaa-f469-439d-91a1-45e0c4aab165-istiod-ca-cert\") pod \"0260aaaa-f469-439d-91a1-45e0c4aab165\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") "
Oct 31 00:49:55.491621 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.491559    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-podinfo\") pod \"0260aaaa-f469-439d-91a1-45e0c4aab165\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") "
Oct 31 00:49:55.491910 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.491674    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-workload-certs\") pod \"0260aaaa-f469-439d-91a1-45e0c4aab165\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") "
Oct 31 00:49:55.491910 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.491758    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-workload-socket\") pod \"0260aaaa-f469-439d-91a1-45e0c4aab165\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") "
Oct 31 00:49:55.491910 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.491786    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-envoy\") pod \"0260aaaa-f469-439d-91a1-45e0c4aab165\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") "
Oct 31 00:49:55.492103 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.491955    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-data\") pod \"0260aaaa-f469-439d-91a1-45e0c4aab165\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") "
Oct 31 00:49:55.492103 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.492001    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0260aaaa-f469-439d-91a1-45e0c4aab165-kube-api-access-c58z2\" (UniqueName: \"kubernetes.io/projected/0260aaaa-f469-439d-91a1-45e0c4aab165-kube-api-access-c58z2\") pod \"0260aaaa-f469-439d-91a1-45e0c4aab165\" (UID: \"0260aaaa-f469-439d-91a1-45e0c4aab165\") "
Oct 31 00:49:55.492336 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.492308    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0260aaaa-f469-439d-91a1-45e0c4aab165-istiod-ca-cert" pod "0260aaaa-f469-439d-91a1-45e0c4aab165" (UID: "0260aaaa-f469-439d-91a1-45e0c4aab165"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:49:55.492806 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.492781    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-workload-socket" pod "0260aaaa-f469-439d-91a1-45e0c4aab165" (UID: "0260aaaa-f469-439d-91a1-45e0c4aab165"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.493063 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.493044    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-credential-socket" pod "0260aaaa-f469-439d-91a1-45e0c4aab165" (UID: "0260aaaa-f469-439d-91a1-45e0c4aab165"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.493179 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.493158    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.493215 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.493188    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.493215 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.493202    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.493274 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.493213    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d509de-a07d-49ba-9a35-67873b87041b-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.493274 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.493225    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d509de-a07d-49ba-9a35-67873b87041b-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.493274 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.493237    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.493274 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.493250    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq8sr\" (UniqueName: \"kubernetes.io/projected/47d509de-a07d-49ba-9a35-67873b87041b-kube-api-access-wq8sr\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.493274 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.493265    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d509de-a07d-49ba-9a35-67873b87041b-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.493274 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.493276    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.493562 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.493288    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d509de-a07d-49ba-9a35-67873b87041b-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.493562 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.493300    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0260aaaa-f469-439d-91a1-45e0c4aab165-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.493562 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.493328    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-workload-certs" pod "0260aaaa-f469-439d-91a1-45e0c4aab165" (UID: "0260aaaa-f469-439d-91a1-45e0c4aab165"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.493562 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.493383    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-data" pod "0260aaaa-f469-439d-91a1-45e0c4aab165" (UID: "0260aaaa-f469-439d-91a1-45e0c4aab165"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.495672 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.495485    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-envoy" pod "0260aaaa-f469-439d-91a1-45e0c4aab165" (UID: "0260aaaa-f469-439d-91a1-45e0c4aab165"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.495672 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.495629    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-token" pod "0260aaaa-f469-439d-91a1-45e0c4aab165" (UID: "0260aaaa-f469-439d-91a1-45e0c4aab165"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:55.495672 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.495414    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0260aaaa-f469-439d-91a1-45e0c4aab165-kube-api-access-c58z2" pod "0260aaaa-f469-439d-91a1-45e0c4aab165" (UID: "0260aaaa-f469-439d-91a1-45e0c4aab165"). InnerVolumeSpecName "kube-api-access-c58z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:55.498053 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.497971    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:49:55.500154 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.500125    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-podinfo" pod "0260aaaa-f469-439d-91a1-45e0c4aab165" (UID: "0260aaaa-f469-439d-91a1-45e0c4aab165"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:49:55.504911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.504829    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:49:55.511592 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.511501    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:49:55.511954 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.511933    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"]
Oct 31 00:49:55.589179 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.572470    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-wqdmj"]
Oct 31 00:49:55.594578 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.594145    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-podinfo\") pod \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") "
Oct 31 00:49:55.594897 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.594877    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-workload-socket\") pod \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") "
Oct 31 00:49:55.595413 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.595298    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-workload-socket" pod "dc70d4a7-43f4-4a03-883b-2035e3f2c137" (UID: "dc70d4a7-43f4-4a03-883b-2035e3f2c137"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.595566 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.595549    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-podinfo\") pod \"80939755-d4c4-4c9e-82dc-6f4349c25415\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") "
Oct 31 00:49:55.595797 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.595776    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-workload-certs\") pod \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") "
Oct 31 00:49:55.596260 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.596142    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-workload-certs" pod "be256a75-3d28-43d7-9b5d-56bf5498f0d1" (UID: "be256a75-3d28-43d7-9b5d-56bf5498f0d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.596826 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.596218    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-token\" (UniqueName: \"kubernetes.io/projected/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-token\") pod \"80939755-d4c4-4c9e-82dc-6f4349c25415\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") "
Oct 31 00:49:55.598044 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.597033    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-podinfo\") pod \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") "
Oct 31 00:49:55.598809 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.598792    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-envoy\") pod \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") "
Oct 31 00:49:55.599243 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.599225    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-data\") pod \"80939755-d4c4-4c9e-82dc-6f4349c25415\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") "
Oct 31 00:49:55.599712 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.599661    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be256a75-3d28-43d7-9b5d-56bf5498f0d1-kube-api-access-qqspt\" (UniqueName: \"kubernetes.io/projected/be256a75-3d28-43d7-9b5d-56bf5498f0d1-kube-api-access-qqspt\") pod \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") "
Oct 31 00:49:55.600195 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.599575    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-data" pod "80939755-d4c4-4c9e-82dc-6f4349c25415" (UID: "80939755-d4c4-4c9e-82dc-6f4349c25415"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.600379 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.600364    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-token\" (UniqueName: \"kubernetes.io/projected/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-token\") pod \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") "
Oct 31 00:49:55.600829 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.600811    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-envoy\") pod \"80939755-d4c4-4c9e-82dc-6f4349c25415\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") "
Oct 31 00:49:55.601035 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.601013    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-workload-socket\") pod \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") "
Oct 31 00:49:55.601196 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.601182    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc70d4a7-43f4-4a03-883b-2035e3f2c137-kube-api-access-zvn2f\" (UniqueName: \"kubernetes.io/projected/dc70d4a7-43f4-4a03-883b-2035e3f2c137-kube-api-access-zvn2f\") pod \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") "
Oct 31 00:49:55.601329 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.601314    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-credential-socket\") pod \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") "
Oct 31 00:49:55.601488 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.601475    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-workload-certs\") pod \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") "
Oct 31 00:49:55.601657 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.601644    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-envoy\") pod \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") "
Oct 31 00:49:55.601868 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.601853    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-data\") pod \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") "
Oct 31 00:49:55.601974 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.601961    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80939755-d4c4-4c9e-82dc-6f4349c25415-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80939755-d4c4-4c9e-82dc-6f4349c25415-istiod-ca-cert\") pod \"80939755-d4c4-4c9e-82dc-6f4349c25415\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") "
Oct 31 00:49:55.602146 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.602132    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-credential-socket\") pod \"80939755-d4c4-4c9e-82dc-6f4349c25415\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") "
Oct 31 00:49:55.602261 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.602246    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istiod-ca-cert\") pod \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") "
Oct 31 00:49:55.602380 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.602367    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-workload-socket\") pod \"80939755-d4c4-4c9e-82dc-6f4349c25415\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") "
Oct 31 00:49:55.602511 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.602497    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istiod-ca-cert\") pod \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") "
Oct 31 00:49:55.602623 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.602609    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-data\") pod \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") "
Oct 31 00:49:55.602763 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.602749    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-token\" (UniqueName: \"kubernetes.io/projected/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-token\") pod \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\" (UID: \"be256a75-3d28-43d7-9b5d-56bf5498f0d1\") "
Oct 31 00:49:55.602891 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.602878    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-credential-socket\") pod \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\" (UID: \"dc70d4a7-43f4-4a03-883b-2035e3f2c137\") "
Oct 31 00:49:55.602992 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.602980    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80939755-d4c4-4c9e-82dc-6f4349c25415-kube-api-access-zvksr\" (UniqueName: \"kubernetes.io/projected/80939755-d4c4-4c9e-82dc-6f4349c25415-kube-api-access-zvksr\") pod \"80939755-d4c4-4c9e-82dc-6f4349c25415\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") "
Oct 31 00:49:55.608403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.605807    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-workload-certs\") pod \"80939755-d4c4-4c9e-82dc-6f4349c25415\" (UID: \"80939755-d4c4-4c9e-82dc-6f4349c25415\") "
Oct 31 00:49:55.608403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.603132    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-podinfo" pod "dc70d4a7-43f4-4a03-883b-2035e3f2c137" (UID: "dc70d4a7-43f4-4a03-883b-2035e3f2c137"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:49:55.608403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.603429    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80939755-d4c4-4c9e-82dc-6f4349c25415-istiod-ca-cert" pod "80939755-d4c4-4c9e-82dc-6f4349c25415" (UID: "80939755-d4c4-4c9e-82dc-6f4349c25415"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:49:55.608403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.604189    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istiod-ca-cert" pod "dc70d4a7-43f4-4a03-883b-2035e3f2c137" (UID: "dc70d4a7-43f4-4a03-883b-2035e3f2c137"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:49:55.608403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.604405    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-credential-socket" pod "80939755-d4c4-4c9e-82dc-6f4349c25415" (UID: "80939755-d4c4-4c9e-82dc-6f4349c25415"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.608403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.604635    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istiod-ca-cert" pod "be256a75-3d28-43d7-9b5d-56bf5498f0d1" (UID: "be256a75-3d28-43d7-9b5d-56bf5498f0d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:49:55.608403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.606903    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be256a75-3d28-43d7-9b5d-56bf5498f0d1-kube-api-access-qqspt" pod "be256a75-3d28-43d7-9b5d-56bf5498f0d1" (UID: "be256a75-3d28-43d7-9b5d-56bf5498f0d1"). InnerVolumeSpecName "kube-api-access-qqspt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:55.608403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.606946    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-envoy" pod "be256a75-3d28-43d7-9b5d-56bf5498f0d1" (UID: "be256a75-3d28-43d7-9b5d-56bf5498f0d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.608403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.607118    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-data" pod "be256a75-3d28-43d7-9b5d-56bf5498f0d1" (UID: "be256a75-3d28-43d7-9b5d-56bf5498f0d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.608403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.607305    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-credential-socket" pod "dc70d4a7-43f4-4a03-883b-2035e3f2c137" (UID: "dc70d4a7-43f4-4a03-883b-2035e3f2c137"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.608403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.607490    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-credential-socket" pod "be256a75-3d28-43d7-9b5d-56bf5498f0d1" (UID: "be256a75-3d28-43d7-9b5d-56bf5498f0d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.608403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.607832    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-workload-socket" pod "80939755-d4c4-4c9e-82dc-6f4349c25415" (UID: "80939755-d4c4-4c9e-82dc-6f4349c25415"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.608403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.608150    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-workload-socket" pod "be256a75-3d28-43d7-9b5d-56bf5498f0d1" (UID: "be256a75-3d28-43d7-9b5d-56bf5498f0d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.608403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.608216    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-workload-certs" pod "80939755-d4c4-4c9e-82dc-6f4349c25415" (UID: "80939755-d4c4-4c9e-82dc-6f4349c25415"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.609143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.608804    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.609143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.608830    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.609143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.608845    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.609143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.608861    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.609143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.608894    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqspt\" (UniqueName: \"kubernetes.io/projected/be256a75-3d28-43d7-9b5d-56bf5498f0d1-kube-api-access-qqspt\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.609143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.608911    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.609143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.608927    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.609143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.608958    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.609143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.608974    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.609143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.608988    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.609143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.609000    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80939755-d4c4-4c9e-82dc-6f4349c25415-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.609143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.609014    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.609143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.609048    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.609143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.609062    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.609143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.609076    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c58z2\" (UniqueName: \"kubernetes.io/projected/0260aaaa-f469-439d-91a1-45e0c4aab165-kube-api-access-c58z2\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.609143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.609092    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.609143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.609106    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.609143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.609119    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.609143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.609132    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.609143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.609145    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.610156 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.609158    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.610156 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.609171    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.610156 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.609186    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0260aaaa-f469-439d-91a1-45e0c4aab165-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.610156 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.609422    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-data" pod "dc70d4a7-43f4-4a03-883b-2035e3f2c137" (UID: "dc70d4a7-43f4-4a03-883b-2035e3f2c137"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.610156 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.609620    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-workload-certs" pod "dc70d4a7-43f4-4a03-883b-2035e3f2c137" (UID: "dc70d4a7-43f4-4a03-883b-2035e3f2c137"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.614092 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.613491    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-podinfo" pod "80939755-d4c4-4c9e-82dc-6f4349c25415" (UID: "80939755-d4c4-4c9e-82dc-6f4349c25415"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:49:55.614092 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.613589    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-token" pod "80939755-d4c4-4c9e-82dc-6f4349c25415" (UID: "80939755-d4c4-4c9e-82dc-6f4349c25415"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:55.614092 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.614048    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-envoy" pod "80939755-d4c4-4c9e-82dc-6f4349c25415" (UID: "80939755-d4c4-4c9e-82dc-6f4349c25415"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.614499 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.614449    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-token" pod "be256a75-3d28-43d7-9b5d-56bf5498f0d1" (UID: "be256a75-3d28-43d7-9b5d-56bf5498f0d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:55.615105 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.615079    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-podinfo" pod "be256a75-3d28-43d7-9b5d-56bf5498f0d1" (UID: "be256a75-3d28-43d7-9b5d-56bf5498f0d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:49:55.616986 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.616948    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc70d4a7-43f4-4a03-883b-2035e3f2c137-kube-api-access-zvn2f" pod "dc70d4a7-43f4-4a03-883b-2035e3f2c137" (UID: "dc70d4a7-43f4-4a03-883b-2035e3f2c137"). InnerVolumeSpecName "kube-api-access-zvn2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:55.658354 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.656918    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-token" pod "dc70d4a7-43f4-4a03-883b-2035e3f2c137" (UID: "dc70d4a7-43f4-4a03-883b-2035e3f2c137"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:55.658354 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.658227    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-envoy" pod "dc70d4a7-43f4-4a03-883b-2035e3f2c137" (UID: "dc70d4a7-43f4-4a03-883b-2035e3f2c137"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.659357 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.659309    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80939755-d4c4-4c9e-82dc-6f4349c25415-kube-api-access-zvksr" pod "80939755-d4c4-4c9e-82dc-6f4349c25415" (UID: "80939755-d4c4-4c9e-82dc-6f4349c25415"). InnerVolumeSpecName "kube-api-access-zvksr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:55.675610 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.675579    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:49:55.682887 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.682814    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:49:55.684994 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.684973    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:49:55.701988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.700259    4327 generic.go:356] "Generic (PLEG): container finished" podID="0260aaaa-f469-439d-91a1-45e0c4aab165" containerID="93d27ffdf13e5fa3cad8c5a67f1f02e8a9a5808c0bfecc5e7b8f829b5f3a3782" exitCode=0
Oct 31 00:49:55.701988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.700347    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x" event={"ID":"0260aaaa-f469-439d-91a1-45e0c4aab165","Type":"ContainerDied","Data":"93d27ffdf13e5fa3cad8c5a67f1f02e8a9a5808c0bfecc5e7b8f829b5f3a3782"}
Oct 31 00:49:55.701988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.700392    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x" event={"ID":"0260aaaa-f469-439d-91a1-45e0c4aab165","Type":"ContainerDied","Data":"b096c64c581906da4c539073078095c0ff3350b64efe6630fb40ed72c58bd3f5"}
Oct 31 00:49:55.701988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.700422    4327 scope.go:122] "RemoveContainer" containerID="93d27ffdf13e5fa3cad8c5a67f1f02e8a9a5808c0bfecc5e7b8f829b5f3a3782"
Oct 31 00:49:55.701988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.700525    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"
Oct 31 00:49:55.709733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.709697    4327 generic.go:356] "Generic (PLEG): container finished" podID="49a4ff5e-3ec9-45ce-88c2-a9b6308a821e" containerID="6ce9aee27c6c3412514e9a07f4333228e5cb3a960b110d3a1cc34ce89476d265" exitCode=0
Oct 31 00:49:55.709873 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.709824    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd" event={"ID":"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e","Type":"ContainerDied","Data":"6ce9aee27c6c3412514e9a07f4333228e5cb3a960b110d3a1cc34ce89476d265"}
Oct 31 00:49:55.709929 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.709873    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd" event={"ID":"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e","Type":"ContainerDied","Data":"ee2a1c7d5700750f28cfa7e15f9b8ba78329bc5798934095e20e11ae3aba4392"}
Oct 31 00:49:55.710024 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.710005    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"
Oct 31 00:49:55.712237 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.712205    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-token\" (UniqueName: \"kubernetes.io/projected/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-token\") pod \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") "
Oct 31 00:49:55.712353 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.712259    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-workload-certs\") pod \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") "
Oct 31 00:49:55.712353 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.712302    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-envoy\") pod \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") "
Oct 31 00:49:55.712473 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.712356    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-credential-socket\") pod \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") "
Oct 31 00:49:55.712473 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.712417    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-token\" (UniqueName: \"kubernetes.io/projected/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-token\") pod \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") "
Oct 31 00:49:55.712584 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.712522    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-workload-socket\") pod \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") "
Oct 31 00:49:55.712632 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.712579    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-credential-socket\") pod \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") "
Oct 31 00:49:55.712723 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.712632    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-kube-api-access-m6r7s\" (UniqueName: \"kubernetes.io/projected/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-kube-api-access-m6r7s\") pod \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") "
Oct 31 00:49:55.712723 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.712665    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-workload-certs\") pod \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") "
Oct 31 00:49:55.713075 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.713050    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-podinfo\") pod \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") "
Oct 31 00:49:55.713182 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.713164    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istiod-ca-cert\") pod \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") "
Oct 31 00:49:55.713235 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.713222    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-kube-api-access-cm2wk\" (UniqueName: \"kubernetes.io/projected/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-kube-api-access-cm2wk\") pod \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") "
Oct 31 00:49:55.713333 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.713316    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-data\") pod \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") "
Oct 31 00:49:55.713395 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.713349    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-envoy\") pod \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") "
Oct 31 00:49:55.713451 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.713409    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istiod-ca-cert\") pod \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") "
Oct 31 00:49:55.713500 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.713463    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-workload-socket\") pod \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\" (UID: \"49a4ff5e-3ec9-45ce-88c2-a9b6308a821e\") "
Oct 31 00:49:55.713616 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.713599    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-podinfo\") pod \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") "
Oct 31 00:49:55.713719 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.713667    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-data\") pod \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\" (UID: \"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f\") "
Oct 31 00:49:55.714616 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.714588    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.714736 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.714621    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvksr\" (UniqueName: \"kubernetes.io/projected/80939755-d4c4-4c9e-82dc-6f4349c25415-kube-api-access-zvksr\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.714736 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.714637    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.714736 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.714652    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.714736 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.714666    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be256a75-3d28-43d7-9b5d-56bf5498f0d1-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.714736 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.714703    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.714736 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.714718    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80939755-d4c4-4c9e-82dc-6f4349c25415-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.714736 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.714733    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvn2f\" (UniqueName: \"kubernetes.io/projected/dc70d4a7-43f4-4a03-883b-2035e3f2c137-kube-api-access-zvn2f\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.715032 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.714746    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.715032 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.714759    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.715032 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.714771    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc70d4a7-43f4-4a03-883b-2035e3f2c137-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.715032 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.715010    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-data" pod "ab534e5f-cfc2-4be6-acf5-bc00c9fa957f" (UID: "ab534e5f-cfc2-4be6-acf5-bc00c9fa957f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.715545 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.715520    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-workload-certs" pod "49a4ff5e-3ec9-45ce-88c2-a9b6308a821e" (UID: "49a4ff5e-3ec9-45ce-88c2-a9b6308a821e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.716458 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.716427    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-data" pod "49a4ff5e-3ec9-45ce-88c2-a9b6308a821e" (UID: "49a4ff5e-3ec9-45ce-88c2-a9b6308a821e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.727536 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.727007    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-credential-socket" pod "49a4ff5e-3ec9-45ce-88c2-a9b6308a821e" (UID: "49a4ff5e-3ec9-45ce-88c2-a9b6308a821e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.727536 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.727276    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istiod-ca-cert" pod "49a4ff5e-3ec9-45ce-88c2-a9b6308a821e" (UID: "49a4ff5e-3ec9-45ce-88c2-a9b6308a821e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:49:55.734427 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.734388    4327 generic.go:356] "Generic (PLEG): container finished" podID="ce1ceaa6-3845-4dc6-8455-974804695d4c" containerID="b9062b534627de3b117cabddc170a59309e3cd8e61152f5b6e47834a3e6cb9da" exitCode=0
Oct 31 00:49:55.734545 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.734517    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2" event={"ID":"ce1ceaa6-3845-4dc6-8455-974804695d4c","Type":"ContainerDied","Data":"b9062b534627de3b117cabddc170a59309e3cd8e61152f5b6e47834a3e6cb9da"}
Oct 31 00:49:55.734606 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.734563    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2" event={"ID":"ce1ceaa6-3845-4dc6-8455-974804695d4c","Type":"ContainerDied","Data":"a5b906315bf935d01500b353442309d1fc456fec7765cb9806513695db392c7f"}
Oct 31 00:49:55.734713 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.734658    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"
Oct 31 00:49:55.753059 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.753022    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-credential-socket" pod "ab534e5f-cfc2-4be6-acf5-bc00c9fa957f" (UID: "ab534e5f-cfc2-4be6-acf5-bc00c9fa957f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.753373 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.753352    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-token" pod "49a4ff5e-3ec9-45ce-88c2-a9b6308a821e" (UID: "49a4ff5e-3ec9-45ce-88c2-a9b6308a821e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:55.756800 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.756769    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-workload-socket" pod "ab534e5f-cfc2-4be6-acf5-bc00c9fa957f" (UID: "ab534e5f-cfc2-4be6-acf5-bc00c9fa957f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.758367 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.758340    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-workload-certs" pod "ab534e5f-cfc2-4be6-acf5-bc00c9fa957f" (UID: "ab534e5f-cfc2-4be6-acf5-bc00c9fa957f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.758650 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.758625    4327 scope.go:122] "RemoveContainer" containerID="93d27ffdf13e5fa3cad8c5a67f1f02e8a9a5808c0bfecc5e7b8f829b5f3a3782"
Oct 31 00:49:55.758865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.758841    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-envoy" pod "49a4ff5e-3ec9-45ce-88c2-a9b6308a821e" (UID: "49a4ff5e-3ec9-45ce-88c2-a9b6308a821e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.759590 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.759568    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-kube-api-access-m6r7s" pod "49a4ff5e-3ec9-45ce-88c2-a9b6308a821e" (UID: "49a4ff5e-3ec9-45ce-88c2-a9b6308a821e"). InnerVolumeSpecName "kube-api-access-m6r7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:55.759935 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.759919    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istiod-ca-cert" pod "ab534e5f-cfc2-4be6-acf5-bc00c9fa957f" (UID: "ab534e5f-cfc2-4be6-acf5-bc00c9fa957f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:49:55.760051 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:49:55.760023    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93d27ffdf13e5fa3cad8c5a67f1f02e8a9a5808c0bfecc5e7b8f829b5f3a3782\": container with ID starting with 93d27ffdf13e5fa3cad8c5a67f1f02e8a9a5808c0bfecc5e7b8f829b5f3a3782 not found: ID does not exist" containerID="93d27ffdf13e5fa3cad8c5a67f1f02e8a9a5808c0bfecc5e7b8f829b5f3a3782"
Oct 31 00:49:55.760104 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.760067    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d27ffdf13e5fa3cad8c5a67f1f02e8a9a5808c0bfecc5e7b8f829b5f3a3782"} err="failed to get container status \"93d27ffdf13e5fa3cad8c5a67f1f02e8a9a5808c0bfecc5e7b8f829b5f3a3782\": rpc error: code = NotFound desc = could not find container \"93d27ffdf13e5fa3cad8c5a67f1f02e8a9a5808c0bfecc5e7b8f829b5f3a3782\": container with ID starting with 93d27ffdf13e5fa3cad8c5a67f1f02e8a9a5808c0bfecc5e7b8f829b5f3a3782 not found: ID does not exist"
Oct 31 00:49:55.760104 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.760093    4327 scope.go:122] "RemoveContainer" containerID="6ce9aee27c6c3412514e9a07f4333228e5cb3a960b110d3a1cc34ce89476d265"
Oct 31 00:49:55.760337 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.760323    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-workload-socket" pod "49a4ff5e-3ec9-45ce-88c2-a9b6308a821e" (UID: "49a4ff5e-3ec9-45ce-88c2-a9b6308a821e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.763199 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.763073    4327 generic.go:356] "Generic (PLEG): container finished" podID="80939755-d4c4-4c9e-82dc-6f4349c25415" containerID="b0c92f0638452a6b2c72c15d0991950d63dbc7fd10ea4e7fe0fc6173ec8178eb" exitCode=0
Oct 31 00:49:55.763199 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.763188    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch" event={"ID":"80939755-d4c4-4c9e-82dc-6f4349c25415","Type":"ContainerDied","Data":"b0c92f0638452a6b2c72c15d0991950d63dbc7fd10ea4e7fe0fc6173ec8178eb"}
Oct 31 00:49:55.763356 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.763228    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"
Oct 31 00:49:55.763356 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.763240    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch" event={"ID":"80939755-d4c4-4c9e-82dc-6f4349c25415","Type":"ContainerDied","Data":"850b065f47c90352250290f1ca8f4c13db4bb2997d82ba49cd1ec2edf9018368"}
Oct 31 00:49:55.763616 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.763598    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-podinfo" pod "ab534e5f-cfc2-4be6-acf5-bc00c9fa957f" (UID: "ab534e5f-cfc2-4be6-acf5-bc00c9fa957f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:49:55.765117 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.765081    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-kube-api-access-cm2wk" pod "ab534e5f-cfc2-4be6-acf5-bc00c9fa957f" (UID: "ab534e5f-cfc2-4be6-acf5-bc00c9fa957f"). InnerVolumeSpecName "kube-api-access-cm2wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:55.769753 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.769725    4327 generic.go:356] "Generic (PLEG): container finished" podID="be256a75-3d28-43d7-9b5d-56bf5498f0d1" containerID="8ae1dd10b9a8f5cb027e10bf10826d754e8bc9dd6bff45b64e005a769c8aa58f" exitCode=0
Oct 31 00:49:55.769850 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.769802    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"
Oct 31 00:49:55.769850 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.769810    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97" event={"ID":"be256a75-3d28-43d7-9b5d-56bf5498f0d1","Type":"ContainerDied","Data":"8ae1dd10b9a8f5cb027e10bf10826d754e8bc9dd6bff45b64e005a769c8aa58f"}
Oct 31 00:49:55.769964 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.769866    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97" event={"ID":"be256a75-3d28-43d7-9b5d-56bf5498f0d1","Type":"ContainerDied","Data":"6b69effa606f03e7f6580f6729a95b82ace24083a89a738192f5a9b436bfe60d"}
Oct 31 00:49:55.770290 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.770247    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-podinfo" pod "49a4ff5e-3ec9-45ce-88c2-a9b6308a821e" (UID: "49a4ff5e-3ec9-45ce-88c2-a9b6308a821e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:49:55.772820 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.772793    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-envoy" pod "ab534e5f-cfc2-4be6-acf5-bc00c9fa957f" (UID: "ab534e5f-cfc2-4be6-acf5-bc00c9fa957f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.774423 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.774400    4327 generic.go:356] "Generic (PLEG): container finished" podID="dc70d4a7-43f4-4a03-883b-2035e3f2c137" containerID="afdc6044c6f2475b0a04c55a3da19a2cef18cc858d5c6902f9ec24bfbd3d6b6b" exitCode=0
Oct 31 00:49:55.774544 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.774521    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr" event={"ID":"dc70d4a7-43f4-4a03-883b-2035e3f2c137","Type":"ContainerDied","Data":"afdc6044c6f2475b0a04c55a3da19a2cef18cc858d5c6902f9ec24bfbd3d6b6b"}
Oct 31 00:49:55.774618 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.774576    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr" event={"ID":"dc70d4a7-43f4-4a03-883b-2035e3f2c137","Type":"ContainerDied","Data":"6e5a74c64e4f5070f3067b5f979a8188c309774b42b54e1bbc8b4ee0f39c1807"}
Oct 31 00:49:55.774673 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.774661    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"
Oct 31 00:49:55.776695 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.776656    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-token" pod "ab534e5f-cfc2-4be6-acf5-bc00c9fa957f" (UID: "ab534e5f-cfc2-4be6-acf5-bc00c9fa957f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:55.779002 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.778982    4327 scope.go:122] "RemoveContainer" containerID="6ce9aee27c6c3412514e9a07f4333228e5cb3a960b110d3a1cc34ce89476d265"
Oct 31 00:49:55.779452 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:49:55.779386    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ce9aee27c6c3412514e9a07f4333228e5cb3a960b110d3a1cc34ce89476d265\": container with ID starting with 6ce9aee27c6c3412514e9a07f4333228e5cb3a960b110d3a1cc34ce89476d265 not found: ID does not exist" containerID="6ce9aee27c6c3412514e9a07f4333228e5cb3a960b110d3a1cc34ce89476d265"
Oct 31 00:49:55.779452 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.779425    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ce9aee27c6c3412514e9a07f4333228e5cb3a960b110d3a1cc34ce89476d265"} err="failed to get container status \"6ce9aee27c6c3412514e9a07f4333228e5cb3a960b110d3a1cc34ce89476d265\": rpc error: code = NotFound desc = could not find container \"6ce9aee27c6c3412514e9a07f4333228e5cb3a960b110d3a1cc34ce89476d265\": container with ID starting with 6ce9aee27c6c3412514e9a07f4333228e5cb3a960b110d3a1cc34ce89476d265 not found: ID does not exist"
Oct 31 00:49:55.779452 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.779449    4327 scope.go:122] "RemoveContainer" containerID="b9062b534627de3b117cabddc170a59309e3cd8e61152f5b6e47834a3e6cb9da"
Oct 31 00:49:55.779758 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.779617    4327 generic.go:356] "Generic (PLEG): container finished" podID="47d509de-a07d-49ba-9a35-67873b87041b" containerID="d783c099b11ddd7cc224d3fa18bc03971543d150ed3069f48f24a36d729b8b2a" exitCode=0
Oct 31 00:49:55.779758 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.779703    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"
Oct 31 00:49:55.779758 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.779750    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj" event={"ID":"47d509de-a07d-49ba-9a35-67873b87041b","Type":"ContainerDied","Data":"d783c099b11ddd7cc224d3fa18bc03971543d150ed3069f48f24a36d729b8b2a"}
Oct 31 00:49:55.782888 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.779800    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj" event={"ID":"47d509de-a07d-49ba-9a35-67873b87041b","Type":"ContainerDied","Data":"543fd442e1bbe418a462a993288f7f0f888cf886b7a08b15cd3132b940df2f48"}
Oct 31 00:49:55.788968 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.788942    4327 generic.go:356] "Generic (PLEG): container finished" podID="ab534e5f-cfc2-4be6-acf5-bc00c9fa957f" containerID="48274dd203ecd370073dda9e8f7e96b8cf8d68fc71f6bc83e57a8e3cef537205" exitCode=0
Oct 31 00:49:55.789050 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.788992    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx" event={"ID":"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f","Type":"ContainerDied","Data":"48274dd203ecd370073dda9e8f7e96b8cf8d68fc71f6bc83e57a8e3cef537205"}
Oct 31 00:49:55.789050 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.789030    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx" event={"ID":"ab534e5f-cfc2-4be6-acf5-bc00c9fa957f","Type":"ContainerDied","Data":"e6bd84c95f2c44264f1b031384b9e70a785124ce7d3f817c308f3f9cc636cf18"}
Oct 31 00:49:55.789119 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.789093    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"
Oct 31 00:49:55.807461 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.807408    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"]
Oct 31 00:49:55.814712 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.814671    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-c6p2x"]
Oct 31 00:49:55.816239 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.816208    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-credential-socket\") pod \"ce1ceaa6-3845-4dc6-8455-974804695d4c\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") "
Oct 31 00:49:55.816321 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.816291    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce1ceaa6-3845-4dc6-8455-974804695d4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1ceaa6-3845-4dc6-8455-974804695d4c-istiod-ca-cert\") pod \"ce1ceaa6-3845-4dc6-8455-974804695d4c\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") "
Oct 31 00:49:55.816364 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.816323    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-podinfo\") pod \"ce1ceaa6-3845-4dc6-8455-974804695d4c\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") "
Oct 31 00:49:55.816400 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.816360    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-envoy\") pod \"ce1ceaa6-3845-4dc6-8455-974804695d4c\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") "
Oct 31 00:49:55.816431 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.816419    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-workload-socket\") pod \"ce1ceaa6-3845-4dc6-8455-974804695d4c\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") "
Oct 31 00:49:55.816556 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.816540    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce1ceaa6-3845-4dc6-8455-974804695d4c-kube-api-access-k2k8l\" (UniqueName: \"kubernetes.io/projected/ce1ceaa6-3845-4dc6-8455-974804695d4c-kube-api-access-k2k8l\") pod \"ce1ceaa6-3845-4dc6-8455-974804695d4c\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") "
Oct 31 00:49:55.816594 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.816579    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-token\" (UniqueName: \"kubernetes.io/projected/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-token\") pod \"ce1ceaa6-3845-4dc6-8455-974804695d4c\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") "
Oct 31 00:49:55.840761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.816776    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-data\") pod \"ce1ceaa6-3845-4dc6-8455-974804695d4c\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") "
Oct 31 00:49:55.840761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.816895    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-workload-certs\") pod \"ce1ceaa6-3845-4dc6-8455-974804695d4c\" (UID: \"ce1ceaa6-3845-4dc6-8455-974804695d4c\") "
Oct 31 00:49:55.840761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.817780    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.840761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.817803    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.840761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.817817    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6r7s\" (UniqueName: \"kubernetes.io/projected/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-kube-api-access-m6r7s\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.840761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.817832    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.840761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.817847    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.840761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.817860    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.840761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.817874    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm2wk\" (UniqueName: \"kubernetes.io/projected/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-kube-api-access-cm2wk\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.840761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.817890    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.840761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.817903    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.840761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.817916    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.840761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.817929    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.840761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.817945    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.840761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.817958    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.840761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.817973    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.840761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.817988    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.840761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.818002    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.840761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.818016    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.840761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.818030    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.844180 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.844135    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-credential-socket" pod "ce1ceaa6-3845-4dc6-8455-974804695d4c" (UID: "ce1ceaa6-3845-4dc6-8455-974804695d4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.845013 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.844989    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-workload-socket" pod "ce1ceaa6-3845-4dc6-8455-974804695d4c" (UID: "ce1ceaa6-3845-4dc6-8455-974804695d4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.847481 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.847458    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-data" pod "ce1ceaa6-3845-4dc6-8455-974804695d4c" (UID: "ce1ceaa6-3845-4dc6-8455-974804695d4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.847753 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.847721    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce1ceaa6-3845-4dc6-8455-974804695d4c-istiod-ca-cert" pod "ce1ceaa6-3845-4dc6-8455-974804695d4c" (UID: "ce1ceaa6-3845-4dc6-8455-974804695d4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:49:55.847991 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.847963    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-workload-certs" pod "ce1ceaa6-3845-4dc6-8455-974804695d4c" (UID: "ce1ceaa6-3845-4dc6-8455-974804695d4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.851579 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.851552    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-token" pod "ce1ceaa6-3845-4dc6-8455-974804695d4c" (UID: "ce1ceaa6-3845-4dc6-8455-974804695d4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:55.851764 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.851660    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1ceaa6-3845-4dc6-8455-974804695d4c-kube-api-access-k2k8l" pod "ce1ceaa6-3845-4dc6-8455-974804695d4c" (UID: "ce1ceaa6-3845-4dc6-8455-974804695d4c"). InnerVolumeSpecName "kube-api-access-k2k8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:55.852227 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.852205    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-envoy" pod "ce1ceaa6-3845-4dc6-8455-974804695d4c" (UID: "ce1ceaa6-3845-4dc6-8455-974804695d4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:55.852944 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.852921    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-podinfo" pod "ce1ceaa6-3845-4dc6-8455-974804695d4c" (UID: "ce1ceaa6-3845-4dc6-8455-974804695d4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:49:55.952658 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.951367    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1ceaa6-3845-4dc6-8455-974804695d4c-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.952658 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.951406    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.952658 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.951421    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.952658 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.951434    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.952658 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.951447    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2k8l\" (UniqueName: \"kubernetes.io/projected/ce1ceaa6-3845-4dc6-8455-974804695d4c-kube-api-access-k2k8l\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.952658 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.951464    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.952658 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.951477    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.952658 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.951491    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.952658 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.951504    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1ceaa6-3845-4dc6-8455-974804695d4c-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:49:55.989009 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.985909    4327 scope.go:122] "RemoveContainer" containerID="b9062b534627de3b117cabddc170a59309e3cd8e61152f5b6e47834a3e6cb9da"
Oct 31 00:49:55.989781 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:49:55.989311    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9062b534627de3b117cabddc170a59309e3cd8e61152f5b6e47834a3e6cb9da\": container with ID starting with b9062b534627de3b117cabddc170a59309e3cd8e61152f5b6e47834a3e6cb9da not found: ID does not exist" containerID="b9062b534627de3b117cabddc170a59309e3cd8e61152f5b6e47834a3e6cb9da"
Oct 31 00:49:55.989781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.989356    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9062b534627de3b117cabddc170a59309e3cd8e61152f5b6e47834a3e6cb9da"} err="failed to get container status \"b9062b534627de3b117cabddc170a59309e3cd8e61152f5b6e47834a3e6cb9da\": rpc error: code = NotFound desc = could not find container \"b9062b534627de3b117cabddc170a59309e3cd8e61152f5b6e47834a3e6cb9da\": container with ID starting with b9062b534627de3b117cabddc170a59309e3cd8e61152f5b6e47834a3e6cb9da not found: ID does not exist"
Oct 31 00:49:55.989781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:55.989384    4327 scope.go:122] "RemoveContainer" containerID="b0c92f0638452a6b2c72c15d0991950d63dbc7fd10ea4e7fe0fc6173ec8178eb"
Oct 31 00:49:56.183506 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.183460    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0260aaaa-f469-439d-91a1-45e0c4aab165" path="/var/lib/kubelet/pods/0260aaaa-f469-439d-91a1-45e0c4aab165/volumes"
Oct 31 00:49:56.185032 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.185004    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="148ce8dc-1145-488d-940e-111d5cb3d9fc" path="/var/lib/kubelet/pods/148ce8dc-1145-488d-940e-111d5cb3d9fc/volumes"
Oct 31 00:49:56.188846 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.188822    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c74e25-811d-414f-b093-a907ca85631d" path="/var/lib/kubelet/pods/c9c74e25-811d-414f-b093-a907ca85631d/volumes"
Oct 31 00:49:56.190274 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.190242    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed99fa70-689b-472e-8e25-62e70c3f2972" path="/var/lib/kubelet/pods/ed99fa70-689b-472e-8e25-62e70c3f2972/volumes"
Oct 31 00:49:56.200264 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.200237    4327 scope.go:122] "RemoveContainer" containerID="b0c92f0638452a6b2c72c15d0991950d63dbc7fd10ea4e7fe0fc6173ec8178eb"
Oct 31 00:49:56.200743 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:49:56.200714    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0c92f0638452a6b2c72c15d0991950d63dbc7fd10ea4e7fe0fc6173ec8178eb\": container with ID starting with b0c92f0638452a6b2c72c15d0991950d63dbc7fd10ea4e7fe0fc6173ec8178eb not found: ID does not exist" containerID="b0c92f0638452a6b2c72c15d0991950d63dbc7fd10ea4e7fe0fc6173ec8178eb"
Oct 31 00:49:56.201167 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.200755    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0c92f0638452a6b2c72c15d0991950d63dbc7fd10ea4e7fe0fc6173ec8178eb"} err="failed to get container status \"b0c92f0638452a6b2c72c15d0991950d63dbc7fd10ea4e7fe0fc6173ec8178eb\": rpc error: code = NotFound desc = could not find container \"b0c92f0638452a6b2c72c15d0991950d63dbc7fd10ea4e7fe0fc6173ec8178eb\": container with ID starting with b0c92f0638452a6b2c72c15d0991950d63dbc7fd10ea4e7fe0fc6173ec8178eb not found: ID does not exist"
Oct 31 00:49:56.201167 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.200781    4327 scope.go:122] "RemoveContainer" containerID="8ae1dd10b9a8f5cb027e10bf10826d754e8bc9dd6bff45b64e005a769c8aa58f"
Oct 31 00:49:56.214238 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.214211    4327 scope.go:122] "RemoveContainer" containerID="8ae1dd10b9a8f5cb027e10bf10826d754e8bc9dd6bff45b64e005a769c8aa58f"
Oct 31 00:49:56.214588 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:49:56.214566    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ae1dd10b9a8f5cb027e10bf10826d754e8bc9dd6bff45b64e005a769c8aa58f\": container with ID starting with 8ae1dd10b9a8f5cb027e10bf10826d754e8bc9dd6bff45b64e005a769c8aa58f not found: ID does not exist" containerID="8ae1dd10b9a8f5cb027e10bf10826d754e8bc9dd6bff45b64e005a769c8aa58f"
Oct 31 00:49:56.214674 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.214597    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ae1dd10b9a8f5cb027e10bf10826d754e8bc9dd6bff45b64e005a769c8aa58f"} err="failed to get container status \"8ae1dd10b9a8f5cb027e10bf10826d754e8bc9dd6bff45b64e005a769c8aa58f\": rpc error: code = NotFound desc = could not find container \"8ae1dd10b9a8f5cb027e10bf10826d754e8bc9dd6bff45b64e005a769c8aa58f\": container with ID starting with 8ae1dd10b9a8f5cb027e10bf10826d754e8bc9dd6bff45b64e005a769c8aa58f not found: ID does not exist"
Oct 31 00:49:56.214674 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.214614    4327 scope.go:122] "RemoveContainer" containerID="afdc6044c6f2475b0a04c55a3da19a2cef18cc858d5c6902f9ec24bfbd3d6b6b"
Oct 31 00:49:56.251311 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.251286    4327 scope.go:122] "RemoveContainer" containerID="afdc6044c6f2475b0a04c55a3da19a2cef18cc858d5c6902f9ec24bfbd3d6b6b"
Oct 31 00:49:56.251803 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:49:56.251774    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afdc6044c6f2475b0a04c55a3da19a2cef18cc858d5c6902f9ec24bfbd3d6b6b\": container with ID starting with afdc6044c6f2475b0a04c55a3da19a2cef18cc858d5c6902f9ec24bfbd3d6b6b not found: ID does not exist" containerID="afdc6044c6f2475b0a04c55a3da19a2cef18cc858d5c6902f9ec24bfbd3d6b6b"
Oct 31 00:49:56.251867 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.251816    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afdc6044c6f2475b0a04c55a3da19a2cef18cc858d5c6902f9ec24bfbd3d6b6b"} err="failed to get container status \"afdc6044c6f2475b0a04c55a3da19a2cef18cc858d5c6902f9ec24bfbd3d6b6b\": rpc error: code = NotFound desc = could not find container \"afdc6044c6f2475b0a04c55a3da19a2cef18cc858d5c6902f9ec24bfbd3d6b6b\": container with ID starting with afdc6044c6f2475b0a04c55a3da19a2cef18cc858d5c6902f9ec24bfbd3d6b6b not found: ID does not exist"
Oct 31 00:49:56.251867 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.251848    4327 scope.go:122] "RemoveContainer" containerID="d783c099b11ddd7cc224d3fa18bc03971543d150ed3069f48f24a36d729b8b2a"
Oct 31 00:49:56.252749 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.252711    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"]
Oct 31 00:49:56.256923 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.256890    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-b7sch"]
Oct 31 00:49:56.263658 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.263634    4327 scope.go:122] "RemoveContainer" containerID="d783c099b11ddd7cc224d3fa18bc03971543d150ed3069f48f24a36d729b8b2a"
Oct 31 00:49:56.264064 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:49:56.264032    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d783c099b11ddd7cc224d3fa18bc03971543d150ed3069f48f24a36d729b8b2a\": container with ID starting with d783c099b11ddd7cc224d3fa18bc03971543d150ed3069f48f24a36d729b8b2a not found: ID does not exist" containerID="d783c099b11ddd7cc224d3fa18bc03971543d150ed3069f48f24a36d729b8b2a"
Oct 31 00:49:56.264156 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.264071    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d783c099b11ddd7cc224d3fa18bc03971543d150ed3069f48f24a36d729b8b2a"} err="failed to get container status \"d783c099b11ddd7cc224d3fa18bc03971543d150ed3069f48f24a36d729b8b2a\": rpc error: code = NotFound desc = could not find container \"d783c099b11ddd7cc224d3fa18bc03971543d150ed3069f48f24a36d729b8b2a\": container with ID starting with d783c099b11ddd7cc224d3fa18bc03971543d150ed3069f48f24a36d729b8b2a not found: ID does not exist"
Oct 31 00:49:56.264156 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.264094    4327 scope.go:122] "RemoveContainer" containerID="48274dd203ecd370073dda9e8f7e96b8cf8d68fc71f6bc83e57a8e3cef537205"
Oct 31 00:49:56.274025 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.273999    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"]
Oct 31 00:49:56.276835 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.276800    4327 scope.go:122] "RemoveContainer" containerID="48274dd203ecd370073dda9e8f7e96b8cf8d68fc71f6bc83e57a8e3cef537205"
Oct 31 00:49:56.277289 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:49:56.277270    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48274dd203ecd370073dda9e8f7e96b8cf8d68fc71f6bc83e57a8e3cef537205\": container with ID starting with 48274dd203ecd370073dda9e8f7e96b8cf8d68fc71f6bc83e57a8e3cef537205 not found: ID does not exist" containerID="48274dd203ecd370073dda9e8f7e96b8cf8d68fc71f6bc83e57a8e3cef537205"
Oct 31 00:49:56.277365 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.277296    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48274dd203ecd370073dda9e8f7e96b8cf8d68fc71f6bc83e57a8e3cef537205"} err="failed to get container status \"48274dd203ecd370073dda9e8f7e96b8cf8d68fc71f6bc83e57a8e3cef537205\": rpc error: code = NotFound desc = could not find container \"48274dd203ecd370073dda9e8f7e96b8cf8d68fc71f6bc83e57a8e3cef537205\": container with ID starting with 48274dd203ecd370073dda9e8f7e96b8cf8d68fc71f6bc83e57a8e3cef537205 not found: ID does not exist"
Oct 31 00:49:56.278708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.278673    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-dclzr"]
Oct 31 00:49:56.312637 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.312606    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"]
Oct 31 00:49:56.317742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.317715    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tfzhd"]
Oct 31 00:49:56.356323 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.356244    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"]
Oct 31 00:49:56.363357 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.363324    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-k6n97"]
Oct 31 00:49:56.398218 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.398185    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"]
Oct 31 00:49:56.448085 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.448048    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-g6tjx"]
Oct 31 00:49:56.455360 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.455313    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"]
Oct 31 00:49:56.459813 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.459778    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-bq6pj"]
Oct 31 00:49:56.490739 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.490701    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"]
Oct 31 00:49:56.496924 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:56.496895    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-gk6l2"]
Oct 31 00:49:57.946326 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:57.946284    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47d509de-a07d-49ba-9a35-67873b87041b" path="/var/lib/kubelet/pods/47d509de-a07d-49ba-9a35-67873b87041b/volumes"
Oct 31 00:49:57.947057 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:57.947035    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49a4ff5e-3ec9-45ce-88c2-a9b6308a821e" path="/var/lib/kubelet/pods/49a4ff5e-3ec9-45ce-88c2-a9b6308a821e/volumes"
Oct 31 00:49:57.947563 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:57.947541    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80939755-d4c4-4c9e-82dc-6f4349c25415" path="/var/lib/kubelet/pods/80939755-d4c4-4c9e-82dc-6f4349c25415/volumes"
Oct 31 00:49:57.948411 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:57.948389    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab534e5f-cfc2-4be6-acf5-bc00c9fa957f" path="/var/lib/kubelet/pods/ab534e5f-cfc2-4be6-acf5-bc00c9fa957f/volumes"
Oct 31 00:49:57.948926 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:57.948906    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be256a75-3d28-43d7-9b5d-56bf5498f0d1" path="/var/lib/kubelet/pods/be256a75-3d28-43d7-9b5d-56bf5498f0d1/volumes"
Oct 31 00:49:57.949400 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:57.949382    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce1ceaa6-3845-4dc6-8455-974804695d4c" path="/var/lib/kubelet/pods/ce1ceaa6-3845-4dc6-8455-974804695d4c/volumes"
Oct 31 00:49:57.950161 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:49:57.950141    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc70d4a7-43f4-4a03-883b-2035e3f2c137" path="/var/lib/kubelet/pods/dc70d4a7-43f4-4a03-883b-2035e3f2c137/volumes"
Oct 31 00:50:01.363275 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.363232    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9"]
Oct 31 00:50:01.364425 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364396    4327 cpu_manager.go:414] "containerMap: removing container" podUID="0260aaaa-f469-439d-91a1-45e0c4aab165" containerName="istio-proxy"
Oct 31 00:50:01.364425 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364425    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0260aaaa-f469-439d-91a1-45e0c4aab165" containerName="istio-proxy"
Oct 31 00:50:01.364581 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364443    4327 cpu_manager.go:414] "containerMap: removing container" podUID="ab534e5f-cfc2-4be6-acf5-bc00c9fa957f" containerName="istio-proxy"
Oct 31 00:50:01.364581 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364452    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab534e5f-cfc2-4be6-acf5-bc00c9fa957f" containerName="istio-proxy"
Oct 31 00:50:01.364581 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364463    4327 cpu_manager.go:414] "containerMap: removing container" podUID="148ce8dc-1145-488d-940e-111d5cb3d9fc" containerName="istio-proxy"
Oct 31 00:50:01.364581 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364470    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="148ce8dc-1145-488d-940e-111d5cb3d9fc" containerName="istio-proxy"
Oct 31 00:50:01.364581 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364505    4327 cpu_manager.go:414] "containerMap: removing container" podUID="49a4ff5e-3ec9-45ce-88c2-a9b6308a821e" containerName="istio-proxy"
Oct 31 00:50:01.364581 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364513    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49a4ff5e-3ec9-45ce-88c2-a9b6308a821e" containerName="istio-proxy"
Oct 31 00:50:01.364581 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364525    4327 cpu_manager.go:414] "containerMap: removing container" podUID="47d509de-a07d-49ba-9a35-67873b87041b" containerName="istio-proxy"
Oct 31 00:50:01.364581 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364533    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47d509de-a07d-49ba-9a35-67873b87041b" containerName="istio-proxy"
Oct 31 00:50:01.364581 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364552    4327 cpu_manager.go:414] "containerMap: removing container" podUID="c9c74e25-811d-414f-b093-a907ca85631d" containerName="istio-proxy"
Oct 31 00:50:01.364581 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364561    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c74e25-811d-414f-b093-a907ca85631d" containerName="istio-proxy"
Oct 31 00:50:01.364581 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364574    4327 cpu_manager.go:414] "containerMap: removing container" podUID="ed99fa70-689b-472e-8e25-62e70c3f2972" containerName="istio-proxy"
Oct 31 00:50:01.364581 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364581    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed99fa70-689b-472e-8e25-62e70c3f2972" containerName="istio-proxy"
Oct 31 00:50:01.364983 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364602    4327 cpu_manager.go:414] "containerMap: removing container" podUID="dc70d4a7-43f4-4a03-883b-2035e3f2c137" containerName="istio-proxy"
Oct 31 00:50:01.364983 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364610    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc70d4a7-43f4-4a03-883b-2035e3f2c137" containerName="istio-proxy"
Oct 31 00:50:01.364983 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364638    4327 cpu_manager.go:414] "containerMap: removing container" podUID="ce1ceaa6-3845-4dc6-8455-974804695d4c" containerName="istio-proxy"
Oct 31 00:50:01.364983 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364645    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce1ceaa6-3845-4dc6-8455-974804695d4c" containerName="istio-proxy"
Oct 31 00:50:01.364983 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364656    4327 cpu_manager.go:414] "containerMap: removing container" podUID="80939755-d4c4-4c9e-82dc-6f4349c25415" containerName="istio-proxy"
Oct 31 00:50:01.364983 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364664    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80939755-d4c4-4c9e-82dc-6f4349c25415" containerName="istio-proxy"
Oct 31 00:50:01.364983 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364699    4327 cpu_manager.go:414] "containerMap: removing container" podUID="be256a75-3d28-43d7-9b5d-56bf5498f0d1" containerName="istio-proxy"
Oct 31 00:50:01.364983 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364707    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be256a75-3d28-43d7-9b5d-56bf5498f0d1" containerName="istio-proxy"
Oct 31 00:50:01.364983 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364898    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="49a4ff5e-3ec9-45ce-88c2-a9b6308a821e" containerName="istio-proxy"
Oct 31 00:50:01.364983 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364913    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="47d509de-a07d-49ba-9a35-67873b87041b" containerName="istio-proxy"
Oct 31 00:50:01.364983 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364926    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c74e25-811d-414f-b093-a907ca85631d" containerName="istio-proxy"
Oct 31 00:50:01.364983 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364936    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed99fa70-689b-472e-8e25-62e70c3f2972" containerName="istio-proxy"
Oct 31 00:50:01.364983 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364949    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc70d4a7-43f4-4a03-883b-2035e3f2c137" containerName="istio-proxy"
Oct 31 00:50:01.364983 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364961    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce1ceaa6-3845-4dc6-8455-974804695d4c" containerName="istio-proxy"
Oct 31 00:50:01.364983 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364972    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="80939755-d4c4-4c9e-82dc-6f4349c25415" containerName="istio-proxy"
Oct 31 00:50:01.364983 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364984    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="be256a75-3d28-43d7-9b5d-56bf5498f0d1" containerName="istio-proxy"
Oct 31 00:50:01.365452 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.364997    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="0260aaaa-f469-439d-91a1-45e0c4aab165" containerName="istio-proxy"
Oct 31 00:50:01.365452 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.365008    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab534e5f-cfc2-4be6-acf5-bc00c9fa957f" containerName="istio-proxy"
Oct 31 00:50:01.365452 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.365016    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="148ce8dc-1145-488d-940e-111d5cb3d9fc" containerName="istio-proxy"
Oct 31 00:50:01.371631 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.371606    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9"
Oct 31 00:50:01.390315 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.390280    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9"]
Oct 31 00:50:01.394687 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.394645    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-3d21c7-serving-cert\""
Oct 31 00:50:01.528205 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.528167    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58153479-6f09-44cf-bdd2-c7eb57580896-metrics-certs\") pod \"maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9\" (UID: \"58153479-6f09-44cf-bdd2-c7eb57580896\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9"
Oct 31 00:50:01.528391 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.528294    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss4q4\" (UniqueName: \"kubernetes.io/projected/58153479-6f09-44cf-bdd2-c7eb57580896-kube-api-access-ss4q4\") pod \"maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9\" (UID: \"58153479-6f09-44cf-bdd2-c7eb57580896\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9"
Oct 31 00:50:01.528491 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.528469    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58153479-6f09-44cf-bdd2-c7eb57580896-maas-api-tls\") pod \"maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9\" (UID: \"58153479-6f09-44cf-bdd2-c7eb57580896\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9"
Oct 31 00:50:01.528716 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.528695    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58153479-6f09-44cf-bdd2-c7eb57580896-openshift-service-ca\") pod \"maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9\" (UID: \"58153479-6f09-44cf-bdd2-c7eb57580896\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9"
Oct 31 00:50:01.652749 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.652525    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss4q4\" (UniqueName: \"kubernetes.io/projected/58153479-6f09-44cf-bdd2-c7eb57580896-kube-api-access-ss4q4\") pod \"maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9\" (UID: \"58153479-6f09-44cf-bdd2-c7eb57580896\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9"
Oct 31 00:50:01.652749 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.652669    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58153479-6f09-44cf-bdd2-c7eb57580896-maas-api-tls\") pod \"maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9\" (UID: \"58153479-6f09-44cf-bdd2-c7eb57580896\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9"
Oct 31 00:50:01.652973 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.652887    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58153479-6f09-44cf-bdd2-c7eb57580896-openshift-service-ca\") pod \"maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9\" (UID: \"58153479-6f09-44cf-bdd2-c7eb57580896\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9"
Oct 31 00:50:01.653024 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.652978    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58153479-6f09-44cf-bdd2-c7eb57580896-metrics-certs\") pod \"maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9\" (UID: \"58153479-6f09-44cf-bdd2-c7eb57580896\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9"
Oct 31 00:50:01.676721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.673585    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58153479-6f09-44cf-bdd2-c7eb57580896-openshift-service-ca\") pod \"maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9\" (UID: \"58153479-6f09-44cf-bdd2-c7eb57580896\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9"
Oct 31 00:50:01.724374 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.722325    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58153479-6f09-44cf-bdd2-c7eb57580896-metrics-certs\") pod \"maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9\" (UID: \"58153479-6f09-44cf-bdd2-c7eb57580896\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9"
Oct 31 00:50:01.724374 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.723273    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss4q4\" (UniqueName: \"kubernetes.io/projected/58153479-6f09-44cf-bdd2-c7eb57580896-kube-api-access-ss4q4\") pod \"maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9\" (UID: \"58153479-6f09-44cf-bdd2-c7eb57580896\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9"
Oct 31 00:50:01.726113 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.725725    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58153479-6f09-44cf-bdd2-c7eb57580896-maas-api-tls\") pod \"maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9\" (UID: \"58153479-6f09-44cf-bdd2-c7eb57580896\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9"
Oct 31 00:50:01.996414 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:01.996321    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9"
Oct 31 00:50:02.336235 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:50:02.336202    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58153479_6f09_44cf_bdd2_c7eb57580896.slice/crio-2a2f96d439ffa74173fa1ec95ee8484cdec2fb75989406f43c19047f46a51ba2 WatchSource:0}: Error finding container 2a2f96d439ffa74173fa1ec95ee8484cdec2fb75989406f43c19047f46a51ba2: Status 404 returned error can't find the container with id 2a2f96d439ffa74173fa1ec95ee8484cdec2fb75989406f43c19047f46a51ba2
Oct 31 00:50:02.336489 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:02.336464    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9"]
Oct 31 00:50:02.338018 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:02.337998    4327 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:50:02.933104 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:02.933068    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9" event={"ID":"58153479-6f09-44cf-bdd2-c7eb57580896","Type":"ContainerStarted","Data":"2a2f96d439ffa74173fa1ec95ee8484cdec2fb75989406f43c19047f46a51ba2"}
Oct 31 00:50:03.942195 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:03.942157    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9" event={"ID":"58153479-6f09-44cf-bdd2-c7eb57580896","Type":"ContainerStarted","Data":"39aaf07926076e15a7c1513db1ab31bddace37e9408cd7cd171a78c96d681797"}
Oct 31 00:50:03.942746 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:03.942347    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9"
Oct 31 00:50:03.994219 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:03.994152    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9" podStartSLOduration=2.424551232 podStartE2EDuration="2.994136202s" totalImagesPullingTime="569.58497ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:03.990161998 +0000 UTC m=+1864.212197544" watchObservedRunningTime="2026-10-31 00:50:03.994136202 +0000 UTC m=+1864.216171744"
Oct 31 00:50:05.116715 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.108761    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"]
Oct 31 00:50:05.116715 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.114309    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.119041 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.119003    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"]
Oct 31 00:50:05.130576 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.130552    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-68f412-openshift-default-dockercfg-77jxq\""
Oct 31 00:50:05.252039 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.251698    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-workload-socket\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.252039 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.251764    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-credential-socket\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.252039 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.251796    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-workload-certs\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.300368 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.252001    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc6mm\" (UniqueName: \"kubernetes.io/projected/400fc179-6148-4af6-b069-88f843600416-kube-api-access-bc6mm\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.300368 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.252398    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/400fc179-6148-4af6-b069-88f843600416-istiod-ca-cert\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.300368 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.252529    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-istio-data\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.300368 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.252615    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/400fc179-6148-4af6-b069-88f843600416-istio-podinfo\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.300368 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.252812    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-istio-envoy\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.300368 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.252904    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/400fc179-6148-4af6-b069-88f843600416-istio-token\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.300368 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.291839    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:50:05.300368 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.291910    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh" podUID="a811bb1a-42b1-4184-b123-c295547116d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:50:05.360834 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.358644    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc6mm\" (UniqueName: \"kubernetes.io/projected/400fc179-6148-4af6-b069-88f843600416-kube-api-access-bc6mm\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.360834 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.358844    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/400fc179-6148-4af6-b069-88f843600416-istiod-ca-cert\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.360834 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.358920    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-istio-data\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.360834 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.359008    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/400fc179-6148-4af6-b069-88f843600416-istio-podinfo\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.360834 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.359069    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-istio-envoy\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.360834 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.359119    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/400fc179-6148-4af6-b069-88f843600416-istio-token\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.360834 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.359175    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-workload-socket\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.360834 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.359216    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-credential-socket\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.360834 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.359247    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-workload-certs\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.366268 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.362907    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/400fc179-6148-4af6-b069-88f843600416-istiod-ca-cert\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.366268 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.363243    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-workload-certs\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.366268 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.363536    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-istio-data\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.366268 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.366090    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-credential-socket\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.366268 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.366226    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-workload-socket\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.377254 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.377179    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/400fc179-6148-4af6-b069-88f843600416-istio-podinfo\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.377464 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.377442    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-istio-envoy\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.387738 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.387655    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc6mm\" (UniqueName: \"kubernetes.io/projected/400fc179-6148-4af6-b069-88f843600416-kube-api-access-bc6mm\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.388936 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.388912    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/400fc179-6148-4af6-b069-88f843600416-istio-token\") pod \"e2e-api-a-68f412-openshift-default-fc857fc67-nccrv\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") " pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.433693 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.433655    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:05.787461 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.787431    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"]
Oct 31 00:50:05.789561 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:50:05.789531    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400fc179_6148_4af6_b069_88f843600416.slice/crio-f0adc72092361bcab543aaf741937164c13dfa5d29cef6863052e28e0d556ecb WatchSource:0}: Error finding container f0adc72092361bcab543aaf741937164c13dfa5d29cef6863052e28e0d556ecb: Status 404 returned error can't find the container with id f0adc72092361bcab543aaf741937164c13dfa5d29cef6863052e28e0d556ecb
Oct 31 00:50:05.793158 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.793118    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:50:05.793270 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.793209    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:50:05.793270 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:05.793256    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:50:06.130072 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:06.130043    4327 generic.go:356] "Generic (PLEG): container finished" podID="a811bb1a-42b1-4184-b123-c295547116d2" containerID="8d72e1c71501630b0c3c7adcfd48ba45fda3046ef9b961815a15a8b58b5adf13" exitCode=0
Oct 31 00:50:06.130522 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:06.130118    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh" event={"ID":"a811bb1a-42b1-4184-b123-c295547116d2","Type":"ContainerDied","Data":"8d72e1c71501630b0c3c7adcfd48ba45fda3046ef9b961815a15a8b58b5adf13"}
Oct 31 00:50:06.130715 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:06.130693    4327 scope.go:122] "RemoveContainer" containerID="8d72e1c71501630b0c3c7adcfd48ba45fda3046ef9b961815a15a8b58b5adf13"
Oct 31 00:50:06.132027 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:06.131987    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv" event={"ID":"400fc179-6148-4af6-b069-88f843600416","Type":"ContainerStarted","Data":"3504921cf5a6184bc5b78a5f156da93de8034956feeb12fd05501d0a3a920905"}
Oct 31 00:50:06.132139 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:06.132051    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv" event={"ID":"400fc179-6148-4af6-b069-88f843600416","Type":"ContainerStarted","Data":"f0adc72092361bcab543aaf741937164c13dfa5d29cef6863052e28e0d556ecb"}
Oct 31 00:50:06.132706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:06.132648    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:50:06.132775 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:06.132735    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:50:06.132775 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:06.132768    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:50:06.225119 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:06.225052    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv" podStartSLOduration=2.225031781 podStartE2EDuration="2.225031781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:06.222186731 +0000 UTC m=+1866.444222296" watchObservedRunningTime="2026-10-31 00:50:06.225031781 +0000 UTC m=+1866.447067324"
Oct 31 00:50:06.437750 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:06.436880    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:06.475705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:06.473080    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:07.161273 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:07.161236    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh" event={"ID":"a811bb1a-42b1-4184-b123-c295547116d2","Type":"ContainerStarted","Data":"4e46f7b1f6bd74edd63f973e1001089653201cceaad28c7ebf1dbfd8abdf0bb4"}
Oct 31 00:50:07.161776 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:07.161705    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:07.162720 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:07.162691    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:50:08.133696 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:08.133648    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:50:08.134919 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:08.134888    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 31 00:50:08.135149 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:08.135121    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh" podUID="a811bb1a-42b1-4184-b123-c295547116d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 31 00:50:09.142036 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:09.138465    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 31 00:50:09.142036 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:09.138541    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh" podUID="a811bb1a-42b1-4184-b123-c295547116d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 31 00:50:10.137704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:10.125212    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 31 00:50:10.137704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:10.125285    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh" podUID="a811bb1a-42b1-4184-b123-c295547116d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 31 00:50:10.144453 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:10.144417    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9"
Oct 31 00:50:10.713598 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:10.713559    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54"]
Oct 31 00:50:10.744480 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:10.744448    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54"
Oct 31 00:50:10.749017 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:10.748984    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54"]
Oct 31 00:50:10.760586 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:10.760558    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-68f412\""
Oct 31 00:50:10.760791 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:10.760769    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-68f412-dockercfg-h8bbx\""
Oct 31 00:50:10.774450 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:10.774408    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15553ed-f60a-42d7-9fea-2daa2a5c5360-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54\" (UID: \"e15553ed-f60a-42d7-9fea-2daa2a5c5360\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54"
Oct 31 00:50:10.774782 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:10.774759    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66tbz\" (UniqueName: \"kubernetes.io/projected/e15553ed-f60a-42d7-9fea-2daa2a5c5360-kube-api-access-66tbz\") pod \"payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54\" (UID: \"e15553ed-f60a-42d7-9fea-2daa2a5c5360\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54"
Oct 31 00:50:10.960783 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:10.911780    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66tbz\" (UniqueName: \"kubernetes.io/projected/e15553ed-f60a-42d7-9fea-2daa2a5c5360-kube-api-access-66tbz\") pod \"payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54\" (UID: \"e15553ed-f60a-42d7-9fea-2daa2a5c5360\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54"
Oct 31 00:50:10.961227 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:10.961194    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15553ed-f60a-42d7-9fea-2daa2a5c5360-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54\" (UID: \"e15553ed-f60a-42d7-9fea-2daa2a5c5360\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54"
Oct 31 00:50:10.982524 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:10.964993    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15553ed-f60a-42d7-9fea-2daa2a5c5360-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54\" (UID: \"e15553ed-f60a-42d7-9fea-2daa2a5c5360\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54"
Oct 31 00:50:11.144703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:11.140366    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 31 00:50:11.227891 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:11.227826    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh" podUID="a811bb1a-42b1-4184-b123-c295547116d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 31 00:50:11.239769 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:11.231457    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66tbz\" (UniqueName: \"kubernetes.io/projected/e15553ed-f60a-42d7-9fea-2daa2a5c5360-kube-api-access-66tbz\") pod \"payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54\" (UID: \"e15553ed-f60a-42d7-9fea-2daa2a5c5360\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54"
Oct 31 00:50:11.239769 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:11.152433    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf"]
Oct 31 00:50:11.298701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:11.287982    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf"
Oct 31 00:50:11.314628 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:11.313817    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf"]
Oct 31 00:50:11.380733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:11.368000    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54"
Oct 31 00:50:11.413201 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:11.413158    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdrgl\" (UniqueName: \"kubernetes.io/projected/99f7b791-5665-432d-bf5d-71a0553094fd-kube-api-access-mdrgl\") pod \"payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf\" (UID: \"99f7b791-5665-432d-bf5d-71a0553094fd\") " pod="openshift-ingress/payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf"
Oct 31 00:50:11.413343 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:11.413327    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99f7b791-5665-432d-bf5d-71a0553094fd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf\" (UID: \"99f7b791-5665-432d-bf5d-71a0553094fd\") " pod="openshift-ingress/payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf"
Oct 31 00:50:11.530749 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:11.523872    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdrgl\" (UniqueName: \"kubernetes.io/projected/99f7b791-5665-432d-bf5d-71a0553094fd-kube-api-access-mdrgl\") pod \"payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf\" (UID: \"99f7b791-5665-432d-bf5d-71a0553094fd\") " pod="openshift-ingress/payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf"
Oct 31 00:50:11.530749 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:11.524022    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99f7b791-5665-432d-bf5d-71a0553094fd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf\" (UID: \"99f7b791-5665-432d-bf5d-71a0553094fd\") " pod="openshift-ingress/payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf"
Oct 31 00:50:11.530749 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:11.525194    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99f7b791-5665-432d-bf5d-71a0553094fd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf\" (UID: \"99f7b791-5665-432d-bf5d-71a0553094fd\") " pod="openshift-ingress/payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf"
Oct 31 00:50:11.561974 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:11.561055    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdrgl\" (UniqueName: \"kubernetes.io/projected/99f7b791-5665-432d-bf5d-71a0553094fd-kube-api-access-mdrgl\") pod \"payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf\" (UID: \"99f7b791-5665-432d-bf5d-71a0553094fd\") " pod="openshift-ingress/payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf"
Oct 31 00:50:11.690125 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:11.690072    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf"
Oct 31 00:50:11.763345 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:11.763314    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54"]
Oct 31 00:50:11.764893 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:50:11.764861    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode15553ed_f60a_42d7_9fea_2daa2a5c5360.slice/crio-4857753e8e4f3d8a67bca1c6397d017cd8dc56afe79a9fbcd58e9bc683094c31 WatchSource:0}: Error finding container 4857753e8e4f3d8a67bca1c6397d017cd8dc56afe79a9fbcd58e9bc683094c31: Status 404 returned error can't find the container with id 4857753e8e4f3d8a67bca1c6397d017cd8dc56afe79a9fbcd58e9bc683094c31
Oct 31 00:50:12.133736 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:12.130084    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 31 00:50:12.133736 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:12.130166    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh" podUID="a811bb1a-42b1-4184-b123-c295547116d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 31 00:50:12.249153 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:50:12.249122    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f7b791_5665_432d_bf5d_71a0553094fd.slice/crio-7dd15d41b2a4586e0f06be770ac9d53ba03f461de7920308f8d0e36dcc05ae36 WatchSource:0}: Error finding container 7dd15d41b2a4586e0f06be770ac9d53ba03f461de7920308f8d0e36dcc05ae36: Status 404 returned error can't find the container with id 7dd15d41b2a4586e0f06be770ac9d53ba03f461de7920308f8d0e36dcc05ae36
Oct 31 00:50:12.249610 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:12.249579    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf"]
Oct 31 00:50:12.355990 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:12.351235    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54" event={"ID":"e15553ed-f60a-42d7-9fea-2daa2a5c5360","Type":"ContainerStarted","Data":"8793a1bcfac933832ee75be879c99e3812c8564af5c117c78c0da91fdf5d03a2"}
Oct 31 00:50:12.355990 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:12.351298    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54" event={"ID":"e15553ed-f60a-42d7-9fea-2daa2a5c5360","Type":"ContainerStarted","Data":"4857753e8e4f3d8a67bca1c6397d017cd8dc56afe79a9fbcd58e9bc683094c31"}
Oct 31 00:50:12.355990 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:12.351328    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54"
Oct 31 00:50:12.355990 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:12.353215    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf" event={"ID":"99f7b791-5665-432d-bf5d-71a0553094fd","Type":"ContainerStarted","Data":"7dd15d41b2a4586e0f06be770ac9d53ba03f461de7920308f8d0e36dcc05ae36"}
Oct 31 00:50:12.397585 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:12.397540    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54" podStartSLOduration=2.397525972 podStartE2EDuration="2.397525972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:12.394240688 +0000 UTC m=+1872.616276233" watchObservedRunningTime="2026-10-31 00:50:12.397525972 +0000 UTC m=+1872.619561557"
Oct 31 00:50:13.130084 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:13.130047    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:50:13.130508 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:13.130478    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:50:13.131421 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:13.131406    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 00:50:13.370457 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:13.370418    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf" event={"ID":"99f7b791-5665-432d-bf5d-71a0553094fd","Type":"ContainerStarted","Data":"949aaea68e12af5240f4a1b6f64c8467772cd4a0a885feb38bc9f9ae34530d90"}
Oct 31 00:50:13.370942 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:13.370527    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf"
Oct 31 00:50:13.438475 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:13.438360    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf" podStartSLOduration=3.438341454 podStartE2EDuration="3.438341454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:13.436192658 +0000 UTC m=+1873.658228200" watchObservedRunningTime="2026-10-31 00:50:13.438341454 +0000 UTC m=+1873.660377001"
Oct 31 00:50:15.296538 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:50:15.296500    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:50:23.375284 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:23.375253    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54"
Oct 31 00:50:24.380904 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:24.380877    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf"
Oct 31 00:50:27.720804 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:27.720752    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5"]
Oct 31 00:50:27.725912 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:27.725885    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5"
Oct 31 00:50:27.773424 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:27.772921    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmn67\" (UniqueName: \"kubernetes.io/projected/a114e7ea-38c3-41c9-8a54-7bcbb47ca591-kube-api-access-rmn67\") pod \"payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5\" (UID: \"a114e7ea-38c3-41c9-8a54-7bcbb47ca591\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5"
Oct 31 00:50:27.773424 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:27.773140    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a114e7ea-38c3-41c9-8a54-7bcbb47ca591-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5\" (UID: \"a114e7ea-38c3-41c9-8a54-7bcbb47ca591\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5"
Oct 31 00:50:27.799701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:27.789217    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-208e9f32\""
Oct 31 00:50:27.799701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:27.789387    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-208e9f32-dockercfg-l7f2c\""
Oct 31 00:50:27.912889 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:27.889328    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a114e7ea-38c3-41c9-8a54-7bcbb47ca591-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5\" (UID: \"a114e7ea-38c3-41c9-8a54-7bcbb47ca591\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5"
Oct 31 00:50:27.912889 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:27.889653    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmn67\" (UniqueName: \"kubernetes.io/projected/a114e7ea-38c3-41c9-8a54-7bcbb47ca591-kube-api-access-rmn67\") pod \"payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5\" (UID: \"a114e7ea-38c3-41c9-8a54-7bcbb47ca591\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5"
Oct 31 00:50:27.912889 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:27.890979    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a114e7ea-38c3-41c9-8a54-7bcbb47ca591-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5\" (UID: \"a114e7ea-38c3-41c9-8a54-7bcbb47ca591\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5"
Oct 31 00:50:28.034754 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:28.001630    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5"]
Oct 31 00:50:28.035061 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:28.031437    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmn67\" (UniqueName: \"kubernetes.io/projected/a114e7ea-38c3-41c9-8a54-7bcbb47ca591-kube-api-access-rmn67\") pod \"payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5\" (UID: \"a114e7ea-38c3-41c9-8a54-7bcbb47ca591\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5"
Oct 31 00:50:28.126949 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:28.126616    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5"
Oct 31 00:50:28.181558 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:28.181526    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc"]
Oct 31 00:50:28.191754 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:28.191727    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc"]
Oct 31 00:50:28.192098 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:28.192080    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc"
Oct 31 00:50:28.209743 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:28.207026    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlfds\" (UniqueName: \"kubernetes.io/projected/67126c73-73e2-40d7-acd8-593ac959261d-kube-api-access-tlfds\") pod \"payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc\" (UID: \"67126c73-73e2-40d7-acd8-593ac959261d\") " pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc"
Oct 31 00:50:28.209743 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:28.207245    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67126c73-73e2-40d7-acd8-593ac959261d-plugins-config-volume\") pod \"payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc\" (UID: \"67126c73-73e2-40d7-acd8-593ac959261d\") " pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc"
Oct 31 00:50:28.308986 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:28.308884    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67126c73-73e2-40d7-acd8-593ac959261d-plugins-config-volume\") pod \"payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc\" (UID: \"67126c73-73e2-40d7-acd8-593ac959261d\") " pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc"
Oct 31 00:50:28.308986 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:28.308967    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlfds\" (UniqueName: \"kubernetes.io/projected/67126c73-73e2-40d7-acd8-593ac959261d-kube-api-access-tlfds\") pod \"payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc\" (UID: \"67126c73-73e2-40d7-acd8-593ac959261d\") " pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc"
Oct 31 00:50:28.309737 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:28.309704    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67126c73-73e2-40d7-acd8-593ac959261d-plugins-config-volume\") pod \"payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc\" (UID: \"67126c73-73e2-40d7-acd8-593ac959261d\") " pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc"
Oct 31 00:50:28.333968 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:28.333939    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlfds\" (UniqueName: \"kubernetes.io/projected/67126c73-73e2-40d7-acd8-593ac959261d-kube-api-access-tlfds\") pod \"payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc\" (UID: \"67126c73-73e2-40d7-acd8-593ac959261d\") " pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc"
Oct 31 00:50:28.504729 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:28.504696    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5"]
Oct 31 00:50:28.504993 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:50:28.504968    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda114e7ea_38c3_41c9_8a54_7bcbb47ca591.slice/crio-3f3fad47baf27473998d0b280343c8c74f4c769f07930a96b39427aebec8e95f WatchSource:0}: Error finding container 3f3fad47baf27473998d0b280343c8c74f4c769f07930a96b39427aebec8e95f: Status 404 returned error can't find the container with id 3f3fad47baf27473998d0b280343c8c74f4c769f07930a96b39427aebec8e95f
Oct 31 00:50:28.546591 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:28.546562    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc"
Oct 31 00:50:28.638969 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:28.638555    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5" event={"ID":"a114e7ea-38c3-41c9-8a54-7bcbb47ca591","Type":"ContainerStarted","Data":"991ea598e0397d613c14a3a976209230c5c3192122d20f287af42577f6c53087"}
Oct 31 00:50:28.638969 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:28.638636    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5" event={"ID":"a114e7ea-38c3-41c9-8a54-7bcbb47ca591","Type":"ContainerStarted","Data":"3f3fad47baf27473998d0b280343c8c74f4c769f07930a96b39427aebec8e95f"}
Oct 31 00:50:28.638969 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:28.638733    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5"
Oct 31 00:50:28.688425 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:28.688355    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5" podStartSLOduration=1.688338754 podStartE2EDuration="1.688338754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:28.682986183 +0000 UTC m=+1888.905021739" watchObservedRunningTime="2026-10-31 00:50:28.688338754 +0000 UTC m=+1888.910374295"
Oct 31 00:50:29.041832 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:29.032619    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc"]
Oct 31 00:50:29.647951 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:29.647907    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc" event={"ID":"67126c73-73e2-40d7-acd8-593ac959261d","Type":"ContainerStarted","Data":"50b5f98b221df7b149d50c9d81a7aeac024f7a2b9a24f9213b7df469bc0e204a"}
Oct 31 00:50:29.648116 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:29.647975    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc" event={"ID":"67126c73-73e2-40d7-acd8-593ac959261d","Type":"ContainerStarted","Data":"246750c9eee220746c55dafa9e57d51317cc1548c4fdde7ff9b00f9909a84cf5"}
Oct 31 00:50:29.700537 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:29.700476    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc" podStartSLOduration=2.700462149 podStartE2EDuration="2.700462149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:29.696617328 +0000 UTC m=+1889.918652872" watchObservedRunningTime="2026-10-31 00:50:29.700462149 +0000 UTC m=+1889.922497691"
Oct 31 00:50:30.656746 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:30.656712    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc"
Oct 31 00:50:32.282778 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:32.278731    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:50:32.402484 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:32.401486    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 00:50:32.402484 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:32.402050    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 00:50:32.417833 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:32.417787    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:50:32.421694 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:32.420857    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:50:33.293457 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:33.279927    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:50:33.293457 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:33.280562    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:50:33.306498 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:33.306465    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 00:50:34.356297 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:34.356254    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"]
Oct 31 00:50:34.356751 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:34.356522    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt" podUID="ee820ea1-a991-40f1-a2f5-b3a9519bc885" containerName="istio-proxy" containerID="cri-o://cc55c21d97152e65dee00c9b2b73adb222c0907d357088d4d7a1751abe64ada3" gracePeriod=30
Oct 31 00:50:34.365635 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:34.365594    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"]
Oct 31 00:50:34.365927 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:34.365892    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz" podUID="9700ab24-a29a-4051-83c7-fc80e83c8ee8" containerName="istio-proxy" containerID="cri-o://a66e34b4b1c659c0dd576a9b2da81dc9bd363591e70e7c22c2e8e8b989847f36" gracePeriod=30
Oct 31 00:50:34.370665 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:34.370640    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"]
Oct 31 00:50:34.370868 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:34.370846    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55" podUID="93505d53-b42a-4c07-ae52-d62ac15f1d79" containerName="istio-proxy" containerID="cri-o://6267638258fe3d5bb055aaeb577dbce11aeeec567f548eb786873ce50e9f9167" gracePeriod=30
Oct 31 00:50:39.653476 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.652194    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5"
Oct 31 00:50:39.695298 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.695259    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:50:39.730652 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.730622    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:50:39.734894 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.734870    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:50:39.855519 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.855478    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-workload-socket\") pod \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") "
Oct 31 00:50:39.855741 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.855535    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-token\" (UniqueName: \"kubernetes.io/projected/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-token\") pod \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") "
Oct 31 00:50:39.855741 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.855588    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-credential-socket\") pod \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") "
Oct 31 00:50:39.855741 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.855628    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93505d53-b42a-4c07-ae52-d62ac15f1d79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93505d53-b42a-4c07-ae52-d62ac15f1d79-istiod-ca-cert\") pod \"93505d53-b42a-4c07-ae52-d62ac15f1d79\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") "
Oct 31 00:50:39.855741 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.855661    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-envoy\") pod \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") "
Oct 31 00:50:39.855741 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.855707    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-envoy\") pod \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") "
Oct 31 00:50:39.856009 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.855777    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-token\" (UniqueName: \"kubernetes.io/projected/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-token\") pod \"93505d53-b42a-4c07-ae52-d62ac15f1d79\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") "
Oct 31 00:50:39.856009 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.855863    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istiod-ca-cert\") pod \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") "
Oct 31 00:50:39.856009 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.855898    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93505d53-b42a-4c07-ae52-d62ac15f1d79-kube-api-access-ll52x\" (UniqueName: \"kubernetes.io/projected/93505d53-b42a-4c07-ae52-d62ac15f1d79-kube-api-access-ll52x\") pod \"93505d53-b42a-4c07-ae52-d62ac15f1d79\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") "
Oct 31 00:50:39.856009 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.855944    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-podinfo\") pod \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") "
Oct 31 00:50:39.856009 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.856008    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9700ab24-a29a-4051-83c7-fc80e83c8ee8-kube-api-access-q8ft9\" (UniqueName: \"kubernetes.io/projected/9700ab24-a29a-4051-83c7-fc80e83c8ee8-kube-api-access-q8ft9\") pod \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") "
Oct 31 00:50:39.856271 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.856089    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-workload-certs\") pod \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") "
Oct 31 00:50:39.856271 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.856122    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-workload-certs\") pod \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") "
Oct 31 00:50:39.856271 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.856156    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istiod-ca-cert\") pod \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") "
Oct 31 00:50:39.856271 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.856236    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-podinfo\") pod \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") "
Oct 31 00:50:39.856442 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.856271    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-credential-socket\") pod \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") "
Oct 31 00:50:39.856442 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.856351    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-data\") pod \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") "
Oct 31 00:50:39.856442 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.856405    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-data\") pod \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") "
Oct 31 00:50:39.856578 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.856490    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-workload-socket\") pod \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\" (UID: \"9700ab24-a29a-4051-83c7-fc80e83c8ee8\") "
Oct 31 00:50:39.856578 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.856572    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-workload-certs\") pod \"93505d53-b42a-4c07-ae52-d62ac15f1d79\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") "
Oct 31 00:50:39.856673 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.856612    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-envoy\") pod \"93505d53-b42a-4c07-ae52-d62ac15f1d79\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") "
Oct 31 00:50:39.856673 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.856654    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee820ea1-a991-40f1-a2f5-b3a9519bc885-kube-api-access-tcfwb\" (UniqueName: \"kubernetes.io/projected/ee820ea1-a991-40f1-a2f5-b3a9519bc885-kube-api-access-tcfwb\") pod \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") "
Oct 31 00:50:39.856803 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.856732    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-data\") pod \"93505d53-b42a-4c07-ae52-d62ac15f1d79\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") "
Oct 31 00:50:39.856803 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.856766    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-token\" (UniqueName: \"kubernetes.io/projected/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-token\") pod \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\" (UID: \"ee820ea1-a991-40f1-a2f5-b3a9519bc885\") "
Oct 31 00:50:39.856927 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.856910    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-credential-socket\") pod \"93505d53-b42a-4c07-ae52-d62ac15f1d79\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") "
Oct 31 00:50:39.856984 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.856943    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-workload-socket\") pod \"93505d53-b42a-4c07-ae52-d62ac15f1d79\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") "
Oct 31 00:50:39.857042 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.856985    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-podinfo\") pod \"93505d53-b42a-4c07-ae52-d62ac15f1d79\" (UID: \"93505d53-b42a-4c07-ae52-d62ac15f1d79\") "
Oct 31 00:50:39.857893 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.857848    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-workload-certs" pod "ee820ea1-a991-40f1-a2f5-b3a9519bc885" (UID: "ee820ea1-a991-40f1-a2f5-b3a9519bc885"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:39.858493 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.858451    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-workload-certs" pod "93505d53-b42a-4c07-ae52-d62ac15f1d79" (UID: "93505d53-b42a-4c07-ae52-d62ac15f1d79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:39.858897 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.858877    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:39.859068 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.859054    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:39.859163 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.859067    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93505d53-b42a-4c07-ae52-d62ac15f1d79-istiod-ca-cert" pod "93505d53-b42a-4c07-ae52-d62ac15f1d79" (UID: "93505d53-b42a-4c07-ae52-d62ac15f1d79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:50:39.859357 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.859331    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-credential-socket" pod "ee820ea1-a991-40f1-a2f5-b3a9519bc885" (UID: "ee820ea1-a991-40f1-a2f5-b3a9519bc885"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:39.859716 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.859696    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istiod-ca-cert" pod "9700ab24-a29a-4051-83c7-fc80e83c8ee8" (UID: "9700ab24-a29a-4051-83c7-fc80e83c8ee8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:50:39.868897 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.868863    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-workload-socket" pod "ee820ea1-a991-40f1-a2f5-b3a9519bc885" (UID: "ee820ea1-a991-40f1-a2f5-b3a9519bc885"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:39.871757 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.871351    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-credential-socket" pod "9700ab24-a29a-4051-83c7-fc80e83c8ee8" (UID: "9700ab24-a29a-4051-83c7-fc80e83c8ee8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:39.871757 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.871503    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-data" pod "ee820ea1-a991-40f1-a2f5-b3a9519bc885" (UID: "ee820ea1-a991-40f1-a2f5-b3a9519bc885"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:39.871757 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.871634    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-data" pod "9700ab24-a29a-4051-83c7-fc80e83c8ee8" (UID: "9700ab24-a29a-4051-83c7-fc80e83c8ee8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:39.871988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.871818    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-workload-socket" pod "9700ab24-a29a-4051-83c7-fc80e83c8ee8" (UID: "9700ab24-a29a-4051-83c7-fc80e83c8ee8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:39.871988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.871893    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-podinfo" pod "ee820ea1-a991-40f1-a2f5-b3a9519bc885" (UID: "ee820ea1-a991-40f1-a2f5-b3a9519bc885"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:50:39.872877 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.872443    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istiod-ca-cert" pod "ee820ea1-a991-40f1-a2f5-b3a9519bc885" (UID: "ee820ea1-a991-40f1-a2f5-b3a9519bc885"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:50:39.872877 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.872567    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-token" pod "9700ab24-a29a-4051-83c7-fc80e83c8ee8" (UID: "9700ab24-a29a-4051-83c7-fc80e83c8ee8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:50:39.872877 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.872581    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-envoy" pod "ee820ea1-a991-40f1-a2f5-b3a9519bc885" (UID: "ee820ea1-a991-40f1-a2f5-b3a9519bc885"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:39.872877 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.872596    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-token" pod "ee820ea1-a991-40f1-a2f5-b3a9519bc885" (UID: "ee820ea1-a991-40f1-a2f5-b3a9519bc885"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:50:39.872877 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.872852    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-data" pod "93505d53-b42a-4c07-ae52-d62ac15f1d79" (UID: "93505d53-b42a-4c07-ae52-d62ac15f1d79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:39.873181 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.872958    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-workload-certs" pod "9700ab24-a29a-4051-83c7-fc80e83c8ee8" (UID: "9700ab24-a29a-4051-83c7-fc80e83c8ee8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:39.873181 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.872989    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-credential-socket" pod "93505d53-b42a-4c07-ae52-d62ac15f1d79" (UID: "93505d53-b42a-4c07-ae52-d62ac15f1d79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:39.873368 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.873326    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-workload-socket" pod "93505d53-b42a-4c07-ae52-d62ac15f1d79" (UID: "93505d53-b42a-4c07-ae52-d62ac15f1d79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:39.873448 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.873381    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-podinfo" pod "9700ab24-a29a-4051-83c7-fc80e83c8ee8" (UID: "9700ab24-a29a-4051-83c7-fc80e83c8ee8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:50:39.873584 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.873555    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93505d53-b42a-4c07-ae52-d62ac15f1d79-kube-api-access-ll52x" pod "93505d53-b42a-4c07-ae52-d62ac15f1d79" (UID: "93505d53-b42a-4c07-ae52-d62ac15f1d79"). InnerVolumeSpecName "kube-api-access-ll52x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:50:39.873776 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.873638    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-token" pod "93505d53-b42a-4c07-ae52-d62ac15f1d79" (UID: "93505d53-b42a-4c07-ae52-d62ac15f1d79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:50:39.873776 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.861562    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-envoy" pod "93505d53-b42a-4c07-ae52-d62ac15f1d79" (UID: "93505d53-b42a-4c07-ae52-d62ac15f1d79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:39.874367 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.874330    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-envoy" pod "9700ab24-a29a-4051-83c7-fc80e83c8ee8" (UID: "9700ab24-a29a-4051-83c7-fc80e83c8ee8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:39.874367 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.874343    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee820ea1-a991-40f1-a2f5-b3a9519bc885-kube-api-access-tcfwb" pod "ee820ea1-a991-40f1-a2f5-b3a9519bc885" (UID: "ee820ea1-a991-40f1-a2f5-b3a9519bc885"). InnerVolumeSpecName "kube-api-access-tcfwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:50:39.875049 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.875010    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-podinfo" pod "93505d53-b42a-4c07-ae52-d62ac15f1d79" (UID: "93505d53-b42a-4c07-ae52-d62ac15f1d79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:50:39.875831 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.875654    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9700ab24-a29a-4051-83c7-fc80e83c8ee8-kube-api-access-q8ft9" pod "9700ab24-a29a-4051-83c7-fc80e83c8ee8" (UID: "9700ab24-a29a-4051-83c7-fc80e83c8ee8"). InnerVolumeSpecName "kube-api-access-q8ft9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:50:39.968113 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.965346    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:39.968113 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.965386    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:39.968781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.968416    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:39.968781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.968453    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:39.968781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.968469    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:39.968781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.968484    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:39.968781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.968497    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93505d53-b42a-4c07-ae52-d62ac15f1d79-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:39.968781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.968511    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:39.968781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.968523    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:39.968781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.968536    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:39.968781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.968549    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:39.968781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.968562    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll52x\" (UniqueName: \"kubernetes.io/projected/93505d53-b42a-4c07-ae52-d62ac15f1d79-kube-api-access-ll52x\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:39.968781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.968576    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:39.968781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.968590    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8ft9\" (UniqueName: \"kubernetes.io/projected/9700ab24-a29a-4051-83c7-fc80e83c8ee8-kube-api-access-q8ft9\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:39.968781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.968603    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:39.968781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.968616    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:39.968781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.968634    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:39.968781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.968646    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:39.968781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.968660    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:39.968781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.968673    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:39.968781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.968702    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9700ab24-a29a-4051-83c7-fc80e83c8ee8-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:39.968781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.968715    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:39.968781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.968730    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcfwb\" (UniqueName: \"kubernetes.io/projected/ee820ea1-a991-40f1-a2f5-b3a9519bc885-kube-api-access-tcfwb\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:39.968781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.968743    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93505d53-b42a-4c07-ae52-d62ac15f1d79-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:39.968781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:39.968756    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee820ea1-a991-40f1-a2f5-b3a9519bc885-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:40.037257 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.037224    4327 generic.go:356] "Generic (PLEG): container finished" podID="ee820ea1-a991-40f1-a2f5-b3a9519bc885" containerID="cc55c21d97152e65dee00c9b2b73adb222c0907d357088d4d7a1751abe64ada3" exitCode=0
Oct 31 00:50:40.037446 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.037287    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"
Oct 31 00:50:40.037446 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.037305    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt" event={"ID":"ee820ea1-a991-40f1-a2f5-b3a9519bc885","Type":"ContainerDied","Data":"cc55c21d97152e65dee00c9b2b73adb222c0907d357088d4d7a1751abe64ada3"}
Oct 31 00:50:40.037446 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.037368    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt" event={"ID":"ee820ea1-a991-40f1-a2f5-b3a9519bc885","Type":"ContainerDied","Data":"1e72807d2434ad479dbbb6fe76e41cccbc2b3b15ddd1f229e4ef2ad05a8ae72f"}
Oct 31 00:50:40.037446 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.037402    4327 scope.go:122] "RemoveContainer" containerID="cc55c21d97152e65dee00c9b2b73adb222c0907d357088d4d7a1751abe64ada3"
Oct 31 00:50:40.039159 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.039129    4327 generic.go:356] "Generic (PLEG): container finished" podID="9700ab24-a29a-4051-83c7-fc80e83c8ee8" containerID="a66e34b4b1c659c0dd576a9b2da81dc9bd363591e70e7c22c2e8e8b989847f36" exitCode=0
Oct 31 00:50:40.039272 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.039239    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz" event={"ID":"9700ab24-a29a-4051-83c7-fc80e83c8ee8","Type":"ContainerDied","Data":"a66e34b4b1c659c0dd576a9b2da81dc9bd363591e70e7c22c2e8e8b989847f36"}
Oct 31 00:50:40.039272 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.039262    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"
Oct 31 00:50:40.039364 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.039285    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz" event={"ID":"9700ab24-a29a-4051-83c7-fc80e83c8ee8","Type":"ContainerDied","Data":"e975d1280d7a35e2c77ff7713f04f44aec4a076c2e1f2fca39dbfef9e1199924"}
Oct 31 00:50:40.041216 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.041196    4327 generic.go:356] "Generic (PLEG): container finished" podID="93505d53-b42a-4c07-ae52-d62ac15f1d79" containerID="6267638258fe3d5bb055aaeb577dbce11aeeec567f548eb786873ce50e9f9167" exitCode=0
Oct 31 00:50:40.041318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.041260    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"
Oct 31 00:50:40.041318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.041295    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55" event={"ID":"93505d53-b42a-4c07-ae52-d62ac15f1d79","Type":"ContainerDied","Data":"6267638258fe3d5bb055aaeb577dbce11aeeec567f548eb786873ce50e9f9167"}
Oct 31 00:50:40.041418 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.041350    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55" event={"ID":"93505d53-b42a-4c07-ae52-d62ac15f1d79","Type":"ContainerDied","Data":"e1cb77d4331d73b3ed66491f2179c82a298e599ecfbcd7e02dac02099d719f27"}
Oct 31 00:50:40.050375 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.050355    4327 scope.go:122] "RemoveContainer" containerID="cc55c21d97152e65dee00c9b2b73adb222c0907d357088d4d7a1751abe64ada3"
Oct 31 00:50:40.050650 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:50:40.050624    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc55c21d97152e65dee00c9b2b73adb222c0907d357088d4d7a1751abe64ada3\": container with ID starting with cc55c21d97152e65dee00c9b2b73adb222c0907d357088d4d7a1751abe64ada3 not found: ID does not exist" containerID="cc55c21d97152e65dee00c9b2b73adb222c0907d357088d4d7a1751abe64ada3"
Oct 31 00:50:40.050750 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.050658    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc55c21d97152e65dee00c9b2b73adb222c0907d357088d4d7a1751abe64ada3"} err="failed to get container status \"cc55c21d97152e65dee00c9b2b73adb222c0907d357088d4d7a1751abe64ada3\": rpc error: code = NotFound desc = could not find container \"cc55c21d97152e65dee00c9b2b73adb222c0907d357088d4d7a1751abe64ada3\": container with ID starting with cc55c21d97152e65dee00c9b2b73adb222c0907d357088d4d7a1751abe64ada3 not found: ID does not exist"
Oct 31 00:50:40.050750 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.050711    4327 scope.go:122] "RemoveContainer" containerID="a66e34b4b1c659c0dd576a9b2da81dc9bd363591e70e7c22c2e8e8b989847f36"
Oct 31 00:50:40.060623 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.060601    4327 scope.go:122] "RemoveContainer" containerID="a66e34b4b1c659c0dd576a9b2da81dc9bd363591e70e7c22c2e8e8b989847f36"
Oct 31 00:50:40.060931 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:50:40.060912    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a66e34b4b1c659c0dd576a9b2da81dc9bd363591e70e7c22c2e8e8b989847f36\": container with ID starting with a66e34b4b1c659c0dd576a9b2da81dc9bd363591e70e7c22c2e8e8b989847f36 not found: ID does not exist" containerID="a66e34b4b1c659c0dd576a9b2da81dc9bd363591e70e7c22c2e8e8b989847f36"
Oct 31 00:50:40.060983 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.060940    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a66e34b4b1c659c0dd576a9b2da81dc9bd363591e70e7c22c2e8e8b989847f36"} err="failed to get container status \"a66e34b4b1c659c0dd576a9b2da81dc9bd363591e70e7c22c2e8e8b989847f36\": rpc error: code = NotFound desc = could not find container \"a66e34b4b1c659c0dd576a9b2da81dc9bd363591e70e7c22c2e8e8b989847f36\": container with ID starting with a66e34b4b1c659c0dd576a9b2da81dc9bd363591e70e7c22c2e8e8b989847f36 not found: ID does not exist"
Oct 31 00:50:40.060983 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.060958    4327 scope.go:122] "RemoveContainer" containerID="6267638258fe3d5bb055aaeb577dbce11aeeec567f548eb786873ce50e9f9167"
Oct 31 00:50:40.073289 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.073259    4327 scope.go:122] "RemoveContainer" containerID="6267638258fe3d5bb055aaeb577dbce11aeeec567f548eb786873ce50e9f9167"
Oct 31 00:50:40.073608 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:50:40.073584    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6267638258fe3d5bb055aaeb577dbce11aeeec567f548eb786873ce50e9f9167\": container with ID starting with 6267638258fe3d5bb055aaeb577dbce11aeeec567f548eb786873ce50e9f9167 not found: ID does not exist" containerID="6267638258fe3d5bb055aaeb577dbce11aeeec567f548eb786873ce50e9f9167"
Oct 31 00:50:40.073806 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.073617    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6267638258fe3d5bb055aaeb577dbce11aeeec567f548eb786873ce50e9f9167"} err="failed to get container status \"6267638258fe3d5bb055aaeb577dbce11aeeec567f548eb786873ce50e9f9167\": rpc error: code = NotFound desc = could not find container \"6267638258fe3d5bb055aaeb577dbce11aeeec567f548eb786873ce50e9f9167\": container with ID starting with 6267638258fe3d5bb055aaeb577dbce11aeeec567f548eb786873ce50e9f9167 not found: ID does not exist"
Oct 31 00:50:40.094030 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.094004    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"]
Oct 31 00:50:40.100929 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.100904    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-mzttt"]
Oct 31 00:50:40.143971 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.143933    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"]
Oct 31 00:50:40.148435 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.148407    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-r5tlz"]
Oct 31 00:50:40.191395 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.191303    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"]
Oct 31 00:50:40.195435 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:40.195407    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-5pv55"]
Oct 31 00:50:41.727972 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:41.727925    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc"
Oct 31 00:50:41.958782 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:41.953741    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93505d53-b42a-4c07-ae52-d62ac15f1d79" path="/var/lib/kubelet/pods/93505d53-b42a-4c07-ae52-d62ac15f1d79/volumes"
Oct 31 00:50:41.958782 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:41.954564    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9700ab24-a29a-4051-83c7-fc80e83c8ee8" path="/var/lib/kubelet/pods/9700ab24-a29a-4051-83c7-fc80e83c8ee8/volumes"
Oct 31 00:50:41.958782 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:41.955285    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee820ea1-a991-40f1-a2f5-b3a9519bc885" path="/var/lib/kubelet/pods/ee820ea1-a991-40f1-a2f5-b3a9519bc885/volumes"
Oct 31 00:50:44.794916 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:44.794822    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5"]
Oct 31 00:50:44.830254 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:44.830221    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4"]
Oct 31 00:50:45.294765 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:50:45.294720    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:50:47.440919 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:47.440879    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr"]
Oct 31 00:50:47.441840 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:47.441814    4327 cpu_manager.go:414] "containerMap: removing container" podUID="ee820ea1-a991-40f1-a2f5-b3a9519bc885" containerName="istio-proxy"
Oct 31 00:50:47.441840 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:47.441838    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee820ea1-a991-40f1-a2f5-b3a9519bc885" containerName="istio-proxy"
Oct 31 00:50:47.442035 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:47.441889    4327 cpu_manager.go:414] "containerMap: removing container" podUID="93505d53-b42a-4c07-ae52-d62ac15f1d79" containerName="istio-proxy"
Oct 31 00:50:47.442035 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:47.441897    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93505d53-b42a-4c07-ae52-d62ac15f1d79" containerName="istio-proxy"
Oct 31 00:50:47.442035 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:47.441922    4327 cpu_manager.go:414] "containerMap: removing container" podUID="9700ab24-a29a-4051-83c7-fc80e83c8ee8" containerName="istio-proxy"
Oct 31 00:50:47.442035 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:47.441928    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9700ab24-a29a-4051-83c7-fc80e83c8ee8" containerName="istio-proxy"
Oct 31 00:50:47.442154 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:47.442038    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="93505d53-b42a-4c07-ae52-d62ac15f1d79" containerName="istio-proxy"
Oct 31 00:50:47.442154 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:47.442057    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="9700ab24-a29a-4051-83c7-fc80e83c8ee8" containerName="istio-proxy"
Oct 31 00:50:47.442154 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:47.442066    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee820ea1-a991-40f1-a2f5-b3a9519bc885" containerName="istio-proxy"
Oct 31 00:50:47.447884 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:47.447867    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr"
Oct 31 00:50:47.465266 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:47.465237    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-68f412-serving-cert\""
Oct 31 00:50:47.471204 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:47.471175    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr"]
Oct 31 00:50:47.501840 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:47.501803    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cd44166-6fbc-4252-919d-42503da5bb55-metrics-certs\") pod \"maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr\" (UID: \"7cd44166-6fbc-4252-919d-42503da5bb55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr"
Oct 31 00:50:47.502060 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:47.502040    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cd44166-6fbc-4252-919d-42503da5bb55-maas-api-tls\") pod \"maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr\" (UID: \"7cd44166-6fbc-4252-919d-42503da5bb55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr"
Oct 31 00:50:47.502105 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:47.502086    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldx46\" (UniqueName: \"kubernetes.io/projected/7cd44166-6fbc-4252-919d-42503da5bb55-kube-api-access-ldx46\") pod \"maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr\" (UID: \"7cd44166-6fbc-4252-919d-42503da5bb55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr"
Oct 31 00:50:47.502205 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:47.502190    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cd44166-6fbc-4252-919d-42503da5bb55-openshift-service-ca\") pod \"maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr\" (UID: \"7cd44166-6fbc-4252-919d-42503da5bb55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr"
Oct 31 00:50:47.603624 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:47.603580    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cd44166-6fbc-4252-919d-42503da5bb55-maas-api-tls\") pod \"maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr\" (UID: \"7cd44166-6fbc-4252-919d-42503da5bb55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr"
Oct 31 00:50:47.603890 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:47.603632    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldx46\" (UniqueName: \"kubernetes.io/projected/7cd44166-6fbc-4252-919d-42503da5bb55-kube-api-access-ldx46\") pod \"maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr\" (UID: \"7cd44166-6fbc-4252-919d-42503da5bb55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr"
Oct 31 00:50:47.603890 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:47.603750    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cd44166-6fbc-4252-919d-42503da5bb55-openshift-service-ca\") pod \"maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr\" (UID: \"7cd44166-6fbc-4252-919d-42503da5bb55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr"
Oct 31 00:50:47.603890 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:47.603842    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cd44166-6fbc-4252-919d-42503da5bb55-metrics-certs\") pod \"maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr\" (UID: \"7cd44166-6fbc-4252-919d-42503da5bb55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr"
Oct 31 00:50:47.604533 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:47.604486    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cd44166-6fbc-4252-919d-42503da5bb55-openshift-service-ca\") pod \"maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr\" (UID: \"7cd44166-6fbc-4252-919d-42503da5bb55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr"
Oct 31 00:50:47.606326 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:47.606305    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cd44166-6fbc-4252-919d-42503da5bb55-maas-api-tls\") pod \"maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr\" (UID: \"7cd44166-6fbc-4252-919d-42503da5bb55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr"
Oct 31 00:50:47.606430 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:47.606356    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cd44166-6fbc-4252-919d-42503da5bb55-metrics-certs\") pod \"maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr\" (UID: \"7cd44166-6fbc-4252-919d-42503da5bb55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr"
Oct 31 00:50:47.629120 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:47.629089    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldx46\" (UniqueName: \"kubernetes.io/projected/7cd44166-6fbc-4252-919d-42503da5bb55-kube-api-access-ldx46\") pod \"maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr\" (UID: \"7cd44166-6fbc-4252-919d-42503da5bb55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr"
Oct 31 00:50:47.766523 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:47.766436    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr"
Oct 31 00:50:48.135923 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:50:48.135889    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd44166_6fbc_4252_919d_42503da5bb55.slice/crio-6213cddf981a29678305db032a2654a7ef6b54b03f1a570c6fe167a117a28527 WatchSource:0}: Error finding container 6213cddf981a29678305db032a2654a7ef6b54b03f1a570c6fe167a117a28527: Status 404 returned error can't find the container with id 6213cddf981a29678305db032a2654a7ef6b54b03f1a570c6fe167a117a28527
Oct 31 00:50:48.136475 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:48.136448    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr"]
Oct 31 00:50:48.241092 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:48.241056    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr" event={"ID":"7cd44166-6fbc-4252-919d-42503da5bb55","Type":"ContainerStarted","Data":"6213cddf981a29678305db032a2654a7ef6b54b03f1a570c6fe167a117a28527"}
Oct 31 00:50:49.250992 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:49.250948    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr" event={"ID":"7cd44166-6fbc-4252-919d-42503da5bb55","Type":"ContainerStarted","Data":"29d4a0b704508f53f3fc1ae36b21d1147c95178a7736787614faf8e85556369a"}
Oct 31 00:50:49.251379 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:49.251009    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr"
Oct 31 00:50:49.345707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:49.342157    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr" podStartSLOduration=2.003788308 podStartE2EDuration="2.342130602s" totalImagesPullingTime="338.342294ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:49.298673283 +0000 UTC m=+1909.520708894" watchObservedRunningTime="2026-10-31 00:50:49.342130602 +0000 UTC m=+1909.564166145"
Oct 31 00:50:49.795984 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:49.795940    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5" podUID="b1822005-c1d3-479b-abb5-0b0f0a4b0cd1" containerName="payload-processing" containerID="cri-o://eee15371f50606cf697279bd22df73dba21b52dc03af3c127bbbf9c958aedc75" gracePeriod=25
Oct 31 00:50:49.830597 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:49.830523    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4" podUID="ddeeaec6-cb86-42c3-9847-ee011c7de8eb" containerName="payload-pre-processing" containerID="cri-o://88faae9fb66f887e93ee3edefc710e4e8fd5135309faf14595c4d0282cc7079d" gracePeriod=25
Oct 31 00:50:49.835469 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:49.835441    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"]
Oct 31 00:50:49.835659 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:49.835639    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5" podUID="dbb151f9-f859-4faf-a830-9da6730837c6" containerName="istio-proxy" containerID="cri-o://074e6cb557d474457106328e29e15a36cadf922763e97e8f81bbec29f7164cc0" gracePeriod=30
Oct 31 00:50:49.848053 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:49.848020    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"]
Oct 31 00:50:49.848294 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:49.848263    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh" podUID="b1acb532-b591-421f-87a7-61fbaec51ed9" containerName="istio-proxy" containerID="cri-o://481c2e052188b2d9ed775b0ff9cca511afbaeedb2ce615a9174dc6da7a1f22ee" gracePeriod=30
Oct 31 00:50:49.853069 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:49.853046    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"]
Oct 31 00:50:49.853254 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:49.853235    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg" podUID="2bd889fe-02e5-45e4-aaa6-156d3beabf00" containerName="istio-proxy" containerID="cri-o://239a4a654071d146230cda27adc28746a87e54465c0bebda4a813aafc7287230" gracePeriod=30
Oct 31 00:50:49.862539 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:49.862509    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"]
Oct 31 00:50:49.862753 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:49.862727    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx" podUID="02dc2389-ff84-4de4-a932-8be187b761e8" containerName="istio-proxy" containerID="cri-o://14b697f9593302afa5dffa82eeaa2cf945b4945a910d8c1a5a8c0379e8d890d9" gracePeriod=30
Oct 31 00:50:50.264148 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:50.264103    4327 generic.go:356] "Generic (PLEG): container finished" podID="ddeeaec6-cb86-42c3-9847-ee011c7de8eb" containerID="88faae9fb66f887e93ee3edefc710e4e8fd5135309faf14595c4d0282cc7079d" exitCode=0
Oct 31 00:50:50.264560 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:50.264228    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4" event={"ID":"ddeeaec6-cb86-42c3-9847-ee011c7de8eb","Type":"ContainerDied","Data":"88faae9fb66f887e93ee3edefc710e4e8fd5135309faf14595c4d0282cc7079d"}
Oct 31 00:50:50.266163 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:50.266136    4327 generic.go:356] "Generic (PLEG): container finished" podID="b1822005-c1d3-479b-abb5-0b0f0a4b0cd1" containerID="eee15371f50606cf697279bd22df73dba21b52dc03af3c127bbbf9c958aedc75" exitCode=0
Oct 31 00:50:50.266288 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:50.266211    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5" event={"ID":"b1822005-c1d3-479b-abb5-0b0f0a4b0cd1","Type":"ContainerDied","Data":"eee15371f50606cf697279bd22df73dba21b52dc03af3c127bbbf9c958aedc75"}
Oct 31 00:50:50.266288 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:50.266269    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5" event={"ID":"b1822005-c1d3-479b-abb5-0b0f0a4b0cd1","Type":"ContainerDied","Data":"2bc7ceabe0c8f9112e8944025e67a63d8053a1ef8de9ca040e0659af2fb0b253"}
Oct 31 00:50:50.266403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:50.266308    4327 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bc7ceabe0c8f9112e8944025e67a63d8053a1ef8de9ca040e0659af2fb0b253"
Oct 31 00:50:50.314906 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:50.314869    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5"
Oct 31 00:50:50.333339 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:50.333305    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1822005-c1d3-479b-abb5-0b0f0a4b0cd1-kube-api-access-vx6v9\" (UniqueName: \"kubernetes.io/projected/b1822005-c1d3-479b-abb5-0b0f0a4b0cd1-kube-api-access-vx6v9\") pod \"b1822005-c1d3-479b-abb5-0b0f0a4b0cd1\" (UID: \"b1822005-c1d3-479b-abb5-0b0f0a4b0cd1\") "
Oct 31 00:50:50.333515 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:50.333355    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1822005-c1d3-479b-abb5-0b0f0a4b0cd1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1822005-c1d3-479b-abb5-0b0f0a4b0cd1-plugins-config-volume\") pod \"b1822005-c1d3-479b-abb5-0b0f0a4b0cd1\" (UID: \"b1822005-c1d3-479b-abb5-0b0f0a4b0cd1\") "
Oct 31 00:50:50.333853 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:50.333825    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1822005-c1d3-479b-abb5-0b0f0a4b0cd1-plugins-config-volume" pod "b1822005-c1d3-479b-abb5-0b0f0a4b0cd1" (UID: "b1822005-c1d3-479b-abb5-0b0f0a4b0cd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:50:50.334891 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:50.334866    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1822005-c1d3-479b-abb5-0b0f0a4b0cd1-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:50.337302 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:50.337273    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1822005-c1d3-479b-abb5-0b0f0a4b0cd1-kube-api-access-vx6v9" pod "b1822005-c1d3-479b-abb5-0b0f0a4b0cd1" (UID: "b1822005-c1d3-479b-abb5-0b0f0a4b0cd1"). InnerVolumeSpecName "kube-api-access-vx6v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:50:50.366616 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:50.366588    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4"
Oct 31 00:50:50.437050 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:50.436961    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddeeaec6-cb86-42c3-9847-ee011c7de8eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddeeaec6-cb86-42c3-9847-ee011c7de8eb-plugins-config-volume\") pod \"ddeeaec6-cb86-42c3-9847-ee011c7de8eb\" (UID: \"ddeeaec6-cb86-42c3-9847-ee011c7de8eb\") "
Oct 31 00:50:50.437205 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:50.437097    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddeeaec6-cb86-42c3-9847-ee011c7de8eb-kube-api-access-4kcc2\" (UniqueName: \"kubernetes.io/projected/ddeeaec6-cb86-42c3-9847-ee011c7de8eb-kube-api-access-4kcc2\") pod \"ddeeaec6-cb86-42c3-9847-ee011c7de8eb\" (UID: \"ddeeaec6-cb86-42c3-9847-ee011c7de8eb\") "
Oct 31 00:50:50.437430 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:50.437398    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddeeaec6-cb86-42c3-9847-ee011c7de8eb-plugins-config-volume" pod "ddeeaec6-cb86-42c3-9847-ee011c7de8eb" (UID: "ddeeaec6-cb86-42c3-9847-ee011c7de8eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:50:50.437906 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:50.437877    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddeeaec6-cb86-42c3-9847-ee011c7de8eb-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:50.437906 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:50.437907    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx6v9\" (UniqueName: \"kubernetes.io/projected/b1822005-c1d3-479b-abb5-0b0f0a4b0cd1-kube-api-access-vx6v9\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:50.439424 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:50.439387    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddeeaec6-cb86-42c3-9847-ee011c7de8eb-kube-api-access-4kcc2" pod "ddeeaec6-cb86-42c3-9847-ee011c7de8eb" (UID: "ddeeaec6-cb86-42c3-9847-ee011c7de8eb"). InnerVolumeSpecName "kube-api-access-4kcc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:50:50.539417 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:50.539372    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kcc2\" (UniqueName: \"kubernetes.io/projected/ddeeaec6-cb86-42c3-9847-ee011c7de8eb-kube-api-access-4kcc2\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:51.275101 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:51.275069    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4"
Oct 31 00:50:51.275525 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:51.275071    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4" event={"ID":"ddeeaec6-cb86-42c3-9847-ee011c7de8eb","Type":"ContainerDied","Data":"22878b09ab1c48308ac1c8a0a918b2cc6a29f889aa0bb90eb6446d46b19c6380"}
Oct 31 00:50:51.275525 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:51.275224    4327 scope.go:122] "RemoveContainer" containerID="88faae9fb66f887e93ee3edefc710e4e8fd5135309faf14595c4d0282cc7079d"
Oct 31 00:50:51.275525 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:51.275376    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5"
Oct 31 00:50:51.329002 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:51.328966    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5"]
Oct 31 00:50:51.341455 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:51.341421    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-14e039-9bcfbb5f9-2hcl5"]
Oct 31 00:50:51.369709 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:51.369640    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4"]
Oct 31 00:50:51.373788 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:51.373761    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-14e039-996548779-dvvg4"]
Oct 31 00:50:51.923792 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:51.923757    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1822005-c1d3-479b-abb5-0b0f0a4b0cd1" path="/var/lib/kubelet/pods/b1822005-c1d3-479b-abb5-0b0f0a4b0cd1/volumes"
Oct 31 00:50:51.924349 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:51.924334    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddeeaec6-cb86-42c3-9847-ee011c7de8eb" path="/var/lib/kubelet/pods/ddeeaec6-cb86-42c3-9847-ee011c7de8eb/volumes"
Oct 31 00:50:54.744176 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:54.744134    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s"]
Oct 31 00:50:54.745272 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:54.745245    4327 cpu_manager.go:414] "containerMap: removing container" podUID="b1822005-c1d3-479b-abb5-0b0f0a4b0cd1" containerName="payload-processing"
Oct 31 00:50:54.745272 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:54.745271    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1822005-c1d3-479b-abb5-0b0f0a4b0cd1" containerName="payload-processing"
Oct 31 00:50:54.745453 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:54.745288    4327 cpu_manager.go:414] "containerMap: removing container" podUID="ddeeaec6-cb86-42c3-9847-ee011c7de8eb" containerName="payload-pre-processing"
Oct 31 00:50:54.745453 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:54.745297    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddeeaec6-cb86-42c3-9847-ee011c7de8eb" containerName="payload-pre-processing"
Oct 31 00:50:54.745562 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:54.745458    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddeeaec6-cb86-42c3-9847-ee011c7de8eb" containerName="payload-pre-processing"
Oct 31 00:50:54.745562 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:54.745495    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1822005-c1d3-479b-abb5-0b0f0a4b0cd1" containerName="payload-processing"
Oct 31 00:50:54.749087 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:54.749058    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s"
Oct 31 00:50:54.772217 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:54.772173    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s"]
Oct 31 00:50:54.778004 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:54.777528    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-208e9f32-serving-cert\""
Oct 31 00:50:54.826788 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:54.826758    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-metrics-certs\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s\" (UID: \"ae6fa740-5a10-4bd4-88cd-51f4006e79f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s"
Oct 31 00:50:54.826788 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:54.826794    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-openshift-service-ca\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s\" (UID: \"ae6fa740-5a10-4bd4-88cd-51f4006e79f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s"
Oct 31 00:50:54.827029 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:54.826922    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-maas-api-tls\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s\" (UID: \"ae6fa740-5a10-4bd4-88cd-51f4006e79f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s"
Oct 31 00:50:54.827029 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:54.826964    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drdwj\" (UniqueName: \"kubernetes.io/projected/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-kube-api-access-drdwj\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s\" (UID: \"ae6fa740-5a10-4bd4-88cd-51f4006e79f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s"
Oct 31 00:50:54.936706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:54.935942    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-metrics-certs\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s\" (UID: \"ae6fa740-5a10-4bd4-88cd-51f4006e79f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s"
Oct 31 00:50:54.936706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:54.936008    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-openshift-service-ca\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s\" (UID: \"ae6fa740-5a10-4bd4-88cd-51f4006e79f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s"
Oct 31 00:50:54.936706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:54.936094    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-maas-api-tls\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s\" (UID: \"ae6fa740-5a10-4bd4-88cd-51f4006e79f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s"
Oct 31 00:50:54.936706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:54.936154    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drdwj\" (UniqueName: \"kubernetes.io/projected/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-kube-api-access-drdwj\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s\" (UID: \"ae6fa740-5a10-4bd4-88cd-51f4006e79f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s"
Oct 31 00:50:54.937053 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:50:54.936905    4327 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-serving-cert: secret "maas-api-e2e-mt-208e9f32-serving-cert" not found
Oct 31 00:50:54.937053 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:50:54.936981    4327 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-maas-api-tls podName:ae6fa740-5a10-4bd4-88cd-51f4006e79f5 nodeName:}" failed. No retries permitted until 2026-10-31 00:50:55.436958015 +0000 UTC m=+1915.658993553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-maas-api-tls") pod "maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s" (UID: "ae6fa740-5a10-4bd4-88cd-51f4006e79f5") : secret "maas-api-e2e-mt-208e9f32-serving-cert" not found
Oct 31 00:50:54.975714 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:54.964267    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-metrics-certs\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s\" (UID: \"ae6fa740-5a10-4bd4-88cd-51f4006e79f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s"
Oct 31 00:50:55.053771 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.031169    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-openshift-service-ca\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s\" (UID: \"ae6fa740-5a10-4bd4-88cd-51f4006e79f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s"
Oct 31 00:50:55.053771 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.038522    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drdwj\" (UniqueName: \"kubernetes.io/projected/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-kube-api-access-drdwj\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s\" (UID: \"ae6fa740-5a10-4bd4-88cd-51f4006e79f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s"
Oct 31 00:50:55.274170 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.274099    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr"
Oct 31 00:50:55.335535 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.335501    4327 generic.go:356] "Generic (PLEG): container finished" podID="02dc2389-ff84-4de4-a932-8be187b761e8" containerID="14b697f9593302afa5dffa82eeaa2cf945b4945a910d8c1a5a8c0379e8d890d9" exitCode=0
Oct 31 00:50:55.335868 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.335773    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx" event={"ID":"02dc2389-ff84-4de4-a932-8be187b761e8","Type":"ContainerDied","Data":"14b697f9593302afa5dffa82eeaa2cf945b4945a910d8c1a5a8c0379e8d890d9"}
Oct 31 00:50:55.338395 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.338316    4327 generic.go:356] "Generic (PLEG): container finished" podID="dbb151f9-f859-4faf-a830-9da6730837c6" containerID="074e6cb557d474457106328e29e15a36cadf922763e97e8f81bbec29f7164cc0" exitCode=0
Oct 31 00:50:55.338513 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.338438    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5" event={"ID":"dbb151f9-f859-4faf-a830-9da6730837c6","Type":"ContainerDied","Data":"074e6cb557d474457106328e29e15a36cadf922763e97e8f81bbec29f7164cc0"}
Oct 31 00:50:55.339934 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.339125    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:50:55.340890 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.340869    4327 generic.go:356] "Generic (PLEG): container finished" podID="b1acb532-b591-421f-87a7-61fbaec51ed9" containerID="481c2e052188b2d9ed775b0ff9cca511afbaeedb2ce615a9174dc6da7a1f22ee" exitCode=0
Oct 31 00:50:55.340989 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.340948    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh" event={"ID":"b1acb532-b591-421f-87a7-61fbaec51ed9","Type":"ContainerDied","Data":"481c2e052188b2d9ed775b0ff9cca511afbaeedb2ce615a9174dc6da7a1f22ee"}
Oct 31 00:50:55.341053 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.340986    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh" event={"ID":"b1acb532-b591-421f-87a7-61fbaec51ed9","Type":"ContainerDied","Data":"0f332d19f9e2ef8b4a1aae74e65bd203249697636980ea765fa7edf18d432829"}
Oct 31 00:50:55.341053 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.341013    4327 scope.go:122] "RemoveContainer" containerID="481c2e052188b2d9ed775b0ff9cca511afbaeedb2ce615a9174dc6da7a1f22ee"
Oct 31 00:50:55.344895 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.344870    4327 generic.go:356] "Generic (PLEG): container finished" podID="2bd889fe-02e5-45e4-aaa6-156d3beabf00" containerID="239a4a654071d146230cda27adc28746a87e54465c0bebda4a813aafc7287230" exitCode=0
Oct 31 00:50:55.345007 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.344946    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg" event={"ID":"2bd889fe-02e5-45e4-aaa6-156d3beabf00","Type":"ContainerDied","Data":"239a4a654071d146230cda27adc28746a87e54465c0bebda4a813aafc7287230"}
Oct 31 00:50:55.362266 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.362239    4327 scope.go:122] "RemoveContainer" containerID="481c2e052188b2d9ed775b0ff9cca511afbaeedb2ce615a9174dc6da7a1f22ee"
Oct 31 00:50:55.362738 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:50:55.362659    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"481c2e052188b2d9ed775b0ff9cca511afbaeedb2ce615a9174dc6da7a1f22ee\": container with ID starting with 481c2e052188b2d9ed775b0ff9cca511afbaeedb2ce615a9174dc6da7a1f22ee not found: ID does not exist" containerID="481c2e052188b2d9ed775b0ff9cca511afbaeedb2ce615a9174dc6da7a1f22ee"
Oct 31 00:50:55.362895 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.362746    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"481c2e052188b2d9ed775b0ff9cca511afbaeedb2ce615a9174dc6da7a1f22ee"} err="failed to get container status \"481c2e052188b2d9ed775b0ff9cca511afbaeedb2ce615a9174dc6da7a1f22ee\": rpc error: code = NotFound desc = could not find container \"481c2e052188b2d9ed775b0ff9cca511afbaeedb2ce615a9174dc6da7a1f22ee\": container with ID starting with 481c2e052188b2d9ed775b0ff9cca511afbaeedb2ce615a9174dc6da7a1f22ee not found: ID does not exist"
Oct 31 00:50:55.383112 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.383087    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:50:55.387373 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.387288    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:50:55.394924 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.394892    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:50:55.454248 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.454209    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1acb532-b591-421f-87a7-61fbaec51ed9-kube-api-access-w69kz\" (UniqueName: \"kubernetes.io/projected/b1acb532-b591-421f-87a7-61fbaec51ed9-kube-api-access-w69kz\") pod \"b1acb532-b591-421f-87a7-61fbaec51ed9\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") "
Oct 31 00:50:55.454452 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.454301    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-token\" (UniqueName: \"kubernetes.io/projected/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-token\") pod \"b1acb532-b591-421f-87a7-61fbaec51ed9\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") "
Oct 31 00:50:55.454452 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.454438    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-data\") pod \"b1acb532-b591-421f-87a7-61fbaec51ed9\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") "
Oct 31 00:50:55.454655 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.454536    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1acb532-b591-421f-87a7-61fbaec51ed9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1acb532-b591-421f-87a7-61fbaec51ed9-istiod-ca-cert\") pod \"b1acb532-b591-421f-87a7-61fbaec51ed9\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") "
Oct 31 00:50:55.454655 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.454578    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-workload-socket\") pod \"b1acb532-b591-421f-87a7-61fbaec51ed9\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") "
Oct 31 00:50:55.454792 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.454655    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-podinfo\") pod \"b1acb532-b591-421f-87a7-61fbaec51ed9\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") "
Oct 31 00:50:55.454792 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.454739    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-envoy\") pod \"b1acb532-b591-421f-87a7-61fbaec51ed9\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") "
Oct 31 00:50:55.454907 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.454826    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-workload-certs\") pod \"b1acb532-b591-421f-87a7-61fbaec51ed9\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") "
Oct 31 00:50:55.454987 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.454969    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-credential-socket\") pod \"b1acb532-b591-421f-87a7-61fbaec51ed9\" (UID: \"b1acb532-b591-421f-87a7-61fbaec51ed9\") "
Oct 31 00:50:55.455240 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.455220    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-maas-api-tls\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s\" (UID: \"ae6fa740-5a10-4bd4-88cd-51f4006e79f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s"
Oct 31 00:50:55.457077 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.457042    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-workload-certs" pod "b1acb532-b591-421f-87a7-61fbaec51ed9" (UID: "b1acb532-b591-421f-87a7-61fbaec51ed9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:55.457944 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.457914    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-workload-socket" pod "b1acb532-b591-421f-87a7-61fbaec51ed9" (UID: "b1acb532-b591-421f-87a7-61fbaec51ed9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:55.458133 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.458111    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-data" pod "b1acb532-b591-421f-87a7-61fbaec51ed9" (UID: "b1acb532-b591-421f-87a7-61fbaec51ed9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:55.458471 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.458445    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1acb532-b591-421f-87a7-61fbaec51ed9-istiod-ca-cert" pod "b1acb532-b591-421f-87a7-61fbaec51ed9" (UID: "b1acb532-b591-421f-87a7-61fbaec51ed9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:50:55.458825 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.458798    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-credential-socket" pod "b1acb532-b591-421f-87a7-61fbaec51ed9" (UID: "b1acb532-b591-421f-87a7-61fbaec51ed9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:55.458925 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.458903    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-envoy" pod "b1acb532-b591-421f-87a7-61fbaec51ed9" (UID: "b1acb532-b591-421f-87a7-61fbaec51ed9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:55.459040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.459013    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-maas-api-tls\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s\" (UID: \"ae6fa740-5a10-4bd4-88cd-51f4006e79f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s"
Oct 31 00:50:55.461005 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.460976    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-podinfo" pod "b1acb532-b591-421f-87a7-61fbaec51ed9" (UID: "b1acb532-b591-421f-87a7-61fbaec51ed9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:50:55.461394 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.461368    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1acb532-b591-421f-87a7-61fbaec51ed9-kube-api-access-w69kz" pod "b1acb532-b591-421f-87a7-61fbaec51ed9" (UID: "b1acb532-b591-421f-87a7-61fbaec51ed9"). InnerVolumeSpecName "kube-api-access-w69kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:50:55.462942 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.462916    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-token" pod "b1acb532-b591-421f-87a7-61fbaec51ed9" (UID: "b1acb532-b591-421f-87a7-61fbaec51ed9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:50:55.556983 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.556899    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02dc2389-ff84-4de4-a932-8be187b761e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02dc2389-ff84-4de4-a932-8be187b761e8-istio-podinfo\") pod \"02dc2389-ff84-4de4-a932-8be187b761e8\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") "
Oct 31 00:50:55.556983 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.556956    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-istio-data\") pod \"02dc2389-ff84-4de4-a932-8be187b761e8\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") "
Oct 31 00:50:55.556983 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.557001    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-istio-data\") pod \"dbb151f9-f859-4faf-a830-9da6730837c6\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") "
Oct 31 00:50:55.557486 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.557035    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-workload-certs\") pod \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") "
Oct 31 00:50:55.557486 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.557069    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-token\" (UniqueName: \"kubernetes.io/projected/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-token\") pod \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") "
Oct 31 00:50:55.557486 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.557130    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-workload-certs\") pod \"02dc2389-ff84-4de4-a932-8be187b761e8\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") "
Oct 31 00:50:55.557486 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.557165    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-workload-socket\") pod \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") "
Oct 31 00:50:55.557486 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.557206    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-workload-socket\") pod \"02dc2389-ff84-4de4-a932-8be187b761e8\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") "
Oct 31 00:50:55.557817 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.557536    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-workload-certs" pod "02dc2389-ff84-4de4-a932-8be187b761e8" (UID: "02dc2389-ff84-4de4-a932-8be187b761e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:55.557817 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.557778    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-workload-socket" pod "2bd889fe-02e5-45e4-aaa6-156d3beabf00" (UID: "2bd889fe-02e5-45e4-aaa6-156d3beabf00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:55.557989 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.557961    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-istio-data" pod "02dc2389-ff84-4de4-a932-8be187b761e8" (UID: "02dc2389-ff84-4de4-a932-8be187b761e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:55.558540 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.558206    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-istio-data" pod "dbb151f9-f859-4faf-a830-9da6730837c6" (UID: "dbb151f9-f859-4faf-a830-9da6730837c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:55.558540 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.558500    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-workload-socket" pod "02dc2389-ff84-4de4-a932-8be187b761e8" (UID: "02dc2389-ff84-4de4-a932-8be187b761e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:55.559068 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.559040    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-workload-certs" pod "2bd889fe-02e5-45e4-aaa6-156d3beabf00" (UID: "2bd889fe-02e5-45e4-aaa6-156d3beabf00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:55.559279 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.559256    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-data\") pod \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") "
Oct 31 00:50:55.561137 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.561117    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-credential-socket\") pod \"dbb151f9-f859-4faf-a830-9da6730837c6\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") "
Oct 31 00:50:55.561259 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.561108    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02dc2389-ff84-4de4-a932-8be187b761e8-istio-podinfo" pod "02dc2389-ff84-4de4-a932-8be187b761e8" (UID: "02dc2389-ff84-4de4-a932-8be187b761e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:50:55.561259 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.561183    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-credential-socket\") pod \"02dc2389-ff84-4de4-a932-8be187b761e8\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") "
Oct 31 00:50:55.561259 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.561212    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-workload-certs\") pod \"dbb151f9-f859-4faf-a830-9da6730837c6\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") "
Oct 31 00:50:55.561259 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.561176    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-data" pod "2bd889fe-02e5-45e4-aaa6-156d3beabf00" (UID: "2bd889fe-02e5-45e4-aaa6-156d3beabf00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:55.561477 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.561262    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-istio-envoy\") pod \"02dc2389-ff84-4de4-a932-8be187b761e8\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") "
Oct 31 00:50:55.561477 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.561346    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02dc2389-ff84-4de4-a932-8be187b761e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02dc2389-ff84-4de4-a932-8be187b761e8-istiod-ca-cert\") pod \"02dc2389-ff84-4de4-a932-8be187b761e8\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") "
Oct 31 00:50:55.561477 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.561367    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-credential-socket" pod "02dc2389-ff84-4de4-a932-8be187b761e8" (UID: "02dc2389-ff84-4de4-a932-8be187b761e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:55.561477 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.561386    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-workload-socket\") pod \"dbb151f9-f859-4faf-a830-9da6730837c6\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") "
Oct 31 00:50:55.561477 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.561403    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-credential-socket" pod "dbb151f9-f859-4faf-a830-9da6730837c6" (UID: "dbb151f9-f859-4faf-a830-9da6730837c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:55.561477 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.561421    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbb151f9-f859-4faf-a830-9da6730837c6-istio-token\" (UniqueName: \"kubernetes.io/projected/dbb151f9-f859-4faf-a830-9da6730837c6-istio-token\") pod \"dbb151f9-f859-4faf-a830-9da6730837c6\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") "
Oct 31 00:50:55.561477 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.561461    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-credential-socket\") pod \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") "
Oct 31 00:50:55.561858 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.561504    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02dc2389-ff84-4de4-a932-8be187b761e8-istio-token\" (UniqueName: \"kubernetes.io/projected/02dc2389-ff84-4de4-a932-8be187b761e8-istio-token\") pod \"02dc2389-ff84-4de4-a932-8be187b761e8\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") "
Oct 31 00:50:55.561858 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.561536    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02dc2389-ff84-4de4-a932-8be187b761e8-kube-api-access-zhm6r\" (UniqueName: \"kubernetes.io/projected/02dc2389-ff84-4de4-a932-8be187b761e8-kube-api-access-zhm6r\") pod \"02dc2389-ff84-4de4-a932-8be187b761e8\" (UID: \"02dc2389-ff84-4de4-a932-8be187b761e8\") "
Oct 31 00:50:55.561858 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.561590    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-podinfo\") pod \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") "
Oct 31 00:50:55.561858 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.561627    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-envoy\") pod \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") "
Oct 31 00:50:55.562406 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.562193    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istiod-ca-cert\") pod \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") "
Oct 31 00:50:55.562406 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.562267    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbb151f9-f859-4faf-a830-9da6730837c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbb151f9-f859-4faf-a830-9da6730837c6-istiod-ca-cert\") pod \"dbb151f9-f859-4faf-a830-9da6730837c6\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") "
Oct 31 00:50:55.562406 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.562302    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbb151f9-f859-4faf-a830-9da6730837c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbb151f9-f859-4faf-a830-9da6730837c6-istio-podinfo\") pod \"dbb151f9-f859-4faf-a830-9da6730837c6\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") "
Oct 31 00:50:55.562406 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.562356    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-istio-envoy\") pod \"dbb151f9-f859-4faf-a830-9da6730837c6\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") "
Oct 31 00:50:55.562791 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.561504    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-workload-certs" pod "dbb151f9-f859-4faf-a830-9da6730837c6" (UID: "dbb151f9-f859-4faf-a830-9da6730837c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:55.562791 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.561772    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02dc2389-ff84-4de4-a932-8be187b761e8-istiod-ca-cert" pod "02dc2389-ff84-4de4-a932-8be187b761e8" (UID: "02dc2389-ff84-4de4-a932-8be187b761e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:50:55.562791 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.561867    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-workload-socket" pod "dbb151f9-f859-4faf-a830-9da6730837c6" (UID: "dbb151f9-f859-4faf-a830-9da6730837c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:55.562791 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.562065    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-credential-socket" pod "2bd889fe-02e5-45e4-aaa6-156d3beabf00" (UID: "2bd889fe-02e5-45e4-aaa6-156d3beabf00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:55.562791 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.562108    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-token" pod "2bd889fe-02e5-45e4-aaa6-156d3beabf00" (UID: "2bd889fe-02e5-45e4-aaa6-156d3beabf00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:50:55.562791 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.562730    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istiod-ca-cert" pod "2bd889fe-02e5-45e4-aaa6-156d3beabf00" (UID: "2bd889fe-02e5-45e4-aaa6-156d3beabf00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:50:55.563157 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.563114    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb151f9-f859-4faf-a830-9da6730837c6-istiod-ca-cert" pod "dbb151f9-f859-4faf-a830-9da6730837c6" (UID: "dbb151f9-f859-4faf-a830-9da6730837c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:50:55.563752 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.563733    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbb151f9-f859-4faf-a830-9da6730837c6-kube-api-access-xbnf5\" (UniqueName: \"kubernetes.io/projected/dbb151f9-f859-4faf-a830-9da6730837c6-kube-api-access-xbnf5\") pod \"dbb151f9-f859-4faf-a830-9da6730837c6\" (UID: \"dbb151f9-f859-4faf-a830-9da6730837c6\") "
Oct 31 00:50:55.564292 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.564274    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd889fe-02e5-45e4-aaa6-156d3beabf00-kube-api-access-qb95m\" (UniqueName: \"kubernetes.io/projected/2bd889fe-02e5-45e4-aaa6-156d3beabf00-kube-api-access-qb95m\") pod \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\" (UID: \"2bd889fe-02e5-45e4-aaa6-156d3beabf00\") "
Oct 31 00:50:55.565361 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.565336    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.565498 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.565482    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.565597 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.565583    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.565873 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.565856    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.566000 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.565966    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w69kz\" (UniqueName: \"kubernetes.io/projected/b1acb532-b591-421f-87a7-61fbaec51ed9-kube-api-access-w69kz\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.566105 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.566069    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-podinfo" pod "2bd889fe-02e5-45e4-aaa6-156d3beabf00" (UID: "2bd889fe-02e5-45e4-aaa6-156d3beabf00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:50:55.566105 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.566095    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.566260 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.566113    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.566260 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.566127    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.566260 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.566141    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.566260 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.566154    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.566260 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.566167    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.566260 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.566180    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02dc2389-ff84-4de4-a932-8be187b761e8-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.566260 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.566194    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1acb532-b591-421f-87a7-61fbaec51ed9-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.566260 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.566206    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.566260 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.566220    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.566260 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.566234    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.566260 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.566247    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.566865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.566285    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.566865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.566301    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbb151f9-f859-4faf-a830-9da6730837c6-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.566865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.566313    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.566865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.566325    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1acb532-b591-421f-87a7-61fbaec51ed9-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.566865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.566337    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02dc2389-ff84-4de4-a932-8be187b761e8-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.566865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.566338    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-envoy" pod "2bd889fe-02e5-45e4-aaa6-156d3beabf00" (UID: "2bd889fe-02e5-45e4-aaa6-156d3beabf00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:55.566865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.566349    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.566865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.566362    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.566865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.566375    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.566865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.566386    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.566865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.566740    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-istio-envoy" pod "02dc2389-ff84-4de4-a932-8be187b761e8" (UID: "02dc2389-ff84-4de4-a932-8be187b761e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:55.568270 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.568052    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbb151f9-f859-4faf-a830-9da6730837c6-istio-podinfo" pod "dbb151f9-f859-4faf-a830-9da6730837c6" (UID: "dbb151f9-f859-4faf-a830-9da6730837c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:50:55.568270 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.568089    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-istio-envoy" pod "dbb151f9-f859-4faf-a830-9da6730837c6" (UID: "dbb151f9-f859-4faf-a830-9da6730837c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:55.568270 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.568190    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb151f9-f859-4faf-a830-9da6730837c6-istio-token" pod "dbb151f9-f859-4faf-a830-9da6730837c6" (UID: "dbb151f9-f859-4faf-a830-9da6730837c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:50:55.568270 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.568220    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02dc2389-ff84-4de4-a932-8be187b761e8-istio-token" pod "02dc2389-ff84-4de4-a932-8be187b761e8" (UID: "02dc2389-ff84-4de4-a932-8be187b761e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:50:55.568643 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.568621    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd889fe-02e5-45e4-aaa6-156d3beabf00-kube-api-access-qb95m" pod "2bd889fe-02e5-45e4-aaa6-156d3beabf00" (UID: "2bd889fe-02e5-45e4-aaa6-156d3beabf00"). InnerVolumeSpecName "kube-api-access-qb95m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:50:55.569051 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.569032    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02dc2389-ff84-4de4-a932-8be187b761e8-kube-api-access-zhm6r" pod "02dc2389-ff84-4de4-a932-8be187b761e8" (UID: "02dc2389-ff84-4de4-a932-8be187b761e8"). InnerVolumeSpecName "kube-api-access-zhm6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:50:55.569952 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.569927    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb151f9-f859-4faf-a830-9da6730837c6-kube-api-access-xbnf5" pod "dbb151f9-f859-4faf-a830-9da6730837c6" (UID: "dbb151f9-f859-4faf-a830-9da6730837c6"). InnerVolumeSpecName "kube-api-access-xbnf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:50:55.686704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.666311    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s"
Oct 31 00:50:55.686704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.669509    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02dc2389-ff84-4de4-a932-8be187b761e8-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.686704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.669542    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbb151f9-f859-4faf-a830-9da6730837c6-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.686704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.669556    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02dc2389-ff84-4de4-a932-8be187b761e8-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.686704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.669571    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhm6r\" (UniqueName: \"kubernetes.io/projected/02dc2389-ff84-4de4-a932-8be187b761e8-kube-api-access-zhm6r\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.686704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.669587    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.686704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.669725    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd889fe-02e5-45e4-aaa6-156d3beabf00-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.686704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.669739    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbb151f9-f859-4faf-a830-9da6730837c6-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.686704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.669751    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbb151f9-f859-4faf-a830-9da6730837c6-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.686704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.669766    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbnf5\" (UniqueName: \"kubernetes.io/projected/dbb151f9-f859-4faf-a830-9da6730837c6-kube-api-access-xbnf5\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:55.686704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:55.669778    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb95m\" (UniqueName: \"kubernetes.io/projected/2bd889fe-02e5-45e4-aaa6-156d3beabf00-kube-api-access-qb95m\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:50:56.240471 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:56.239850    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s"]
Oct 31 00:50:56.247403 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:50:56.245898    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6fa740_5a10_4bd4_88cd_51f4006e79f5.slice/crio-843eaef1a3a4d43aac4ffac9974edee9583c3dc14d9cf6adb1edb8ddf9b32d51 WatchSource:0}: Error finding container 843eaef1a3a4d43aac4ffac9974edee9583c3dc14d9cf6adb1edb8ddf9b32d51: Status 404 returned error can't find the container with id 843eaef1a3a4d43aac4ffac9974edee9583c3dc14d9cf6adb1edb8ddf9b32d51
Oct 31 00:50:56.271083 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:56.271049    4327 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:50:56.271292 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:56.271110    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5" podUID="dbb151f9-f859-4faf-a830-9da6730837c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:50:56.358609 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:56.358579    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"
Oct 31 00:50:56.358830 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:56.358613    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx" event={"ID":"02dc2389-ff84-4de4-a932-8be187b761e8","Type":"ContainerDied","Data":"e43a1490fcfbf86a6784cdeeaa269743efc56f57ac44dabc0fba4f340da251b0"}
Oct 31 00:50:56.358830 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:56.358696    4327 scope.go:122] "RemoveContainer" containerID="14b697f9593302afa5dffa82eeaa2cf945b4945a910d8c1a5a8c0379e8d890d9"
Oct 31 00:50:56.360796 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:56.360241    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"
Oct 31 00:50:56.360796 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:56.360264    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5" event={"ID":"dbb151f9-f859-4faf-a830-9da6730837c6","Type":"ContainerDied","Data":"06d0d477b5f29b5621123fef99b2a122f7b5828381a2ff0f9d6a38589ae985ab"}
Oct 31 00:50:56.361486 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:56.361468    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"
Oct 31 00:50:56.363385 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:56.363367    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"
Oct 31 00:50:56.363480 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:56.363411    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg" event={"ID":"2bd889fe-02e5-45e4-aaa6-156d3beabf00","Type":"ContainerDied","Data":"bff79a93dfed6c13fb8cdd6ef86923fe40af48887d0af30967e4c4b9d77559f2"}
Oct 31 00:50:56.364947 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:56.364920    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s" event={"ID":"ae6fa740-5a10-4bd4-88cd-51f4006e79f5","Type":"ContainerStarted","Data":"843eaef1a3a4d43aac4ffac9974edee9583c3dc14d9cf6adb1edb8ddf9b32d51"}
Oct 31 00:50:56.371168 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:56.371151    4327 scope.go:122] "RemoveContainer" containerID="074e6cb557d474457106328e29e15a36cadf922763e97e8f81bbec29f7164cc0"
Oct 31 00:50:56.381532 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:56.381510    4327 scope.go:122] "RemoveContainer" containerID="239a4a654071d146230cda27adc28746a87e54465c0bebda4a813aafc7287230"
Oct 31 00:50:56.416526 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:56.416492    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"]
Oct 31 00:50:56.428774 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:56.428739    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-d5vwh"]
Oct 31 00:50:56.454771 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:56.454733    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"]
Oct 31 00:50:56.461392 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:56.461356    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-v82wg"]
Oct 31 00:50:56.498048 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:56.497956    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"]
Oct 31 00:50:56.519757 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:56.517862    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-r6rs5"]
Oct 31 00:50:56.556282 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:56.556237    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"]
Oct 31 00:50:56.562734 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:56.562656    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-hgkfx"]
Oct 31 00:50:57.375505 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:57.375464    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s" event={"ID":"ae6fa740-5a10-4bd4-88cd-51f4006e79f5","Type":"ContainerStarted","Data":"374cd93c64cfa4778de289d6fe1b02af6f28b87f13cd6552f2bb443ecb82209a"}
Oct 31 00:50:57.376046 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:57.375537    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s"
Oct 31 00:50:57.422664 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:57.422601    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s" podStartSLOduration=3.079341366 podStartE2EDuration="3.422582024s" totalImagesPullingTime="343.240658ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:57.418187072 +0000 UTC m=+1917.640222616" watchObservedRunningTime="2026-10-31 00:50:57.422582024 +0000 UTC m=+1917.644617570"
Oct 31 00:50:57.922914 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:57.922875    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02dc2389-ff84-4de4-a932-8be187b761e8" path="/var/lib/kubelet/pods/02dc2389-ff84-4de4-a932-8be187b761e8/volumes"
Oct 31 00:50:57.923560 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:57.923537    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bd889fe-02e5-45e4-aaa6-156d3beabf00" path="/var/lib/kubelet/pods/2bd889fe-02e5-45e4-aaa6-156d3beabf00/volumes"
Oct 31 00:50:57.924141 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:57.924119    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1acb532-b591-421f-87a7-61fbaec51ed9" path="/var/lib/kubelet/pods/b1acb532-b591-421f-87a7-61fbaec51ed9/volumes"
Oct 31 00:50:57.925037 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:50:57.925018    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbb151f9-f859-4faf-a830-9da6730837c6" path="/var/lib/kubelet/pods/dbb151f9-f859-4faf-a830-9da6730837c6/volumes"
Oct 31 00:51:03.401741 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:03.401709    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s"
Oct 31 00:51:07.327598 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:07.327551    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq"]
Oct 31 00:51:07.370810 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:07.370766    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm"]
Oct 31 00:51:10.941766 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:10.941727    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"]
Oct 31 00:51:10.942484 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:10.942467    4327 cpu_manager.go:414] "containerMap: removing container" podUID="dbb151f9-f859-4faf-a830-9da6730837c6" containerName="istio-proxy"
Oct 31 00:51:10.942484 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:10.942484    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbb151f9-f859-4faf-a830-9da6730837c6" containerName="istio-proxy"
Oct 31 00:51:10.942612 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:10.942509    4327 cpu_manager.go:414] "containerMap: removing container" podUID="b1acb532-b591-421f-87a7-61fbaec51ed9" containerName="istio-proxy"
Oct 31 00:51:10.942612 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:10.942515    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1acb532-b591-421f-87a7-61fbaec51ed9" containerName="istio-proxy"
Oct 31 00:51:10.942612 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:10.942533    4327 cpu_manager.go:414] "containerMap: removing container" podUID="2bd889fe-02e5-45e4-aaa6-156d3beabf00" containerName="istio-proxy"
Oct 31 00:51:10.942612 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:10.942538    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd889fe-02e5-45e4-aaa6-156d3beabf00" containerName="istio-proxy"
Oct 31 00:51:10.942612 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:10.942574    4327 cpu_manager.go:414] "containerMap: removing container" podUID="02dc2389-ff84-4de4-a932-8be187b761e8" containerName="istio-proxy"
Oct 31 00:51:10.942612 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:10.942580    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02dc2389-ff84-4de4-a932-8be187b761e8" containerName="istio-proxy"
Oct 31 00:51:10.942951 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:10.942670    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1acb532-b591-421f-87a7-61fbaec51ed9" containerName="istio-proxy"
Oct 31 00:51:10.942951 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:10.942706    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd889fe-02e5-45e4-aaa6-156d3beabf00" containerName="istio-proxy"
Oct 31 00:51:10.942951 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:10.942719    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="02dc2389-ff84-4de4-a932-8be187b761e8" containerName="istio-proxy"
Oct 31 00:51:10.942951 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:10.942725    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbb151f9-f859-4faf-a830-9da6730837c6" containerName="istio-proxy"
Oct 31 00:51:10.948354 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:10.948333    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:10.986623 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:10.986585    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"]
Oct 31 00:51:10.992954 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:10.992927    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4d5621-openshift-default-dockercfg-hn752\""
Oct 31 00:51:11.016027 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.015986    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z829t\" (UniqueName: \"kubernetes.io/projected/e30c1c23-92c5-47b6-a250-3519dbfae94b-kube-api-access-z829t\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.016208 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.016076    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-workload-certs\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.016208 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.016108    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-credential-socket\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.016357 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.016260    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-token\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.016357 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.016319    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-workload-socket\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.016469 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.016436    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-data\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.016705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.016539    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-envoy\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.016705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.016619    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e30c1c23-92c5-47b6-a250-3519dbfae94b-istiod-ca-cert\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.016705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.016696    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-podinfo\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.118079 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.118039    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-data\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.118273 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.118161    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-envoy\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.118273 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.118242    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e30c1c23-92c5-47b6-a250-3519dbfae94b-istiod-ca-cert\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.118407 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.118303    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-podinfo\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.118407 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.118340    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z829t\" (UniqueName: \"kubernetes.io/projected/e30c1c23-92c5-47b6-a250-3519dbfae94b-kube-api-access-z829t\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.118522 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.118405    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-workload-certs\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.118522 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.118418    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-data\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.118522 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.118433    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-credential-socket\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.118715 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.118580    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-token\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.118715 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.118647    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-workload-socket\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.118821 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.118740    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-credential-socket\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.118872 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.118820    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-workload-certs\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.119030 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.119013    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-workload-socket\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.119077 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.119046    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e30c1c23-92c5-47b6-a250-3519dbfae94b-istiod-ca-cert\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.120516 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.120495    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-envoy\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.120777 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.120758    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-podinfo\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.189395 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.188816    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-token\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.241754 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.213100    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z829t\" (UniqueName: \"kubernetes.io/projected/e30c1c23-92c5-47b6-a250-3519dbfae94b-kube-api-access-z829t\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.269350 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.269004    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:11.744782 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.744660    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"]
Oct 31 00:51:11.750458 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.750435    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:11.769465 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.769416    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"]
Oct 31 00:51:11.775729 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.775645    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-bbb902-openshift-default-dockercfg-vbp54\""
Oct 31 00:51:11.782101 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.782065    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:51:11.782229 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.782152    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:51:11.782229 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.782197    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:51:11.783127 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.783105    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"]
Oct 31 00:51:11.871021 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.870990    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-data\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:11.871191 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.871093    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-token\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:11.871191 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.871165    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-credential-socket\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:11.871455 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.871434    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-envoy\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:11.871520 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.871478    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-workload-certs\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:11.871520 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.871496    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-podinfo\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:11.871707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.871662    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-workload-socket\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:11.871822 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.871797    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ed7530-493b-48d4-ac03-a4d9179504ff-istiod-ca-cert\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:11.871874 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.871847    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79stl\" (UniqueName: \"kubernetes.io/projected/46ed7530-493b-48d4-ac03-a4d9179504ff-kube-api-access-79stl\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:11.983702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.977259    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-envoy\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:11.983702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.977342    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-workload-certs\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:11.983702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.977375    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-podinfo\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:11.983702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.977463    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-workload-socket\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:11.983702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.977577    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ed7530-493b-48d4-ac03-a4d9179504ff-istiod-ca-cert\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:11.983702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.977619    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79stl\" (UniqueName: \"kubernetes.io/projected/46ed7530-493b-48d4-ac03-a4d9179504ff-kube-api-access-79stl\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:11.983702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.977672    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-data\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:11.983702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.977733    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-token\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:11.983702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.977779    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-credential-socket\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:11.983702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.978532    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-credential-socket\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:11.985701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.985257    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-workload-certs\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:11.985701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.985432    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ed7530-493b-48d4-ac03-a4d9179504ff-istiod-ca-cert\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:11.989696 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.986135    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-data\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:11.989696 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:11.986969    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-workload-socket\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:12.036757 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.029904    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-envoy\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:12.038040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.037969    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-podinfo\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:12.055716 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.041457    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79stl\" (UniqueName: \"kubernetes.io/projected/46ed7530-493b-48d4-ac03-a4d9179504ff-kube-api-access-79stl\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:12.062701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.056714    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-token\") pod \"e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") " pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:12.087298 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.087006    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:12.340693 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.340443    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq" podUID="9f9893fc-b75a-4bed-925f-c02168ffc76e" containerName="payload-processing" containerID="cri-o://ead9d4cbb81ca58e22d70c3854a6714287e3d59d5ae07275a99ea41853dad039" gracePeriod=25
Oct 31 00:51:12.371504 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.371439    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm" podUID="19b21c31-d906-4652-81e1-765d08fc3e5e" containerName="payload-pre-processing" containerID="cri-o://5f85f06de98a67b708f185f3df346b475c9242e2ff724c3dd41a20ca4982b9ef" gracePeriod=25
Oct 31 00:51:12.567409 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.566875    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht" event={"ID":"e30c1c23-92c5-47b6-a250-3519dbfae94b","Type":"ContainerStarted","Data":"ab5032d092a1c46a9a13905c1fd3dd5df965157cba66865ffc90ace375d5a265"}
Oct 31 00:51:12.567409 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.566940    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht" event={"ID":"e30c1c23-92c5-47b6-a250-3519dbfae94b","Type":"ContainerStarted","Data":"47dc15b9ee081433899a048539535a56388168370630d1534df69f4b1fb37e8d"}
Oct 31 00:51:12.571984 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.571945    4327 generic.go:356] "Generic (PLEG): container finished" podID="19b21c31-d906-4652-81e1-765d08fc3e5e" containerID="5f85f06de98a67b708f185f3df346b475c9242e2ff724c3dd41a20ca4982b9ef" exitCode=0
Oct 31 00:51:12.572152 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.572051    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm" event={"ID":"19b21c31-d906-4652-81e1-765d08fc3e5e","Type":"ContainerDied","Data":"5f85f06de98a67b708f185f3df346b475c9242e2ff724c3dd41a20ca4982b9ef"}
Oct 31 00:51:12.574637 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.574611    4327 generic.go:356] "Generic (PLEG): container finished" podID="9f9893fc-b75a-4bed-925f-c02168ffc76e" containerID="ead9d4cbb81ca58e22d70c3854a6714287e3d59d5ae07275a99ea41853dad039" exitCode=0
Oct 31 00:51:12.574785 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.574737    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq" event={"ID":"9f9893fc-b75a-4bed-925f-c02168ffc76e","Type":"ContainerDied","Data":"ead9d4cbb81ca58e22d70c3854a6714287e3d59d5ae07275a99ea41853dad039"}
Oct 31 00:51:12.682385 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.682356    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq"
Oct 31 00:51:12.715923 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.714859    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht" podStartSLOduration=2.714837037 podStartE2EDuration="2.714837037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:12.619615688 +0000 UTC m=+1932.841651231" watchObservedRunningTime="2026-10-31 00:51:12.714837037 +0000 UTC m=+1932.936872581"
Oct 31 00:51:12.719169 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.719134    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"]
Oct 31 00:51:12.721194 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:51:12.720973    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ed7530_493b_48d4_ac03_a4d9179504ff.slice/crio-96217f6a42053d3f979e847414deb9bf1d66776677fb4ea114ee62886db2b747 WatchSource:0}: Error finding container 96217f6a42053d3f979e847414deb9bf1d66776677fb4ea114ee62886db2b747: Status 404 returned error can't find the container with id 96217f6a42053d3f979e847414deb9bf1d66776677fb4ea114ee62886db2b747
Oct 31 00:51:12.723715 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.723644    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:51:12.725395 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.723767    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:51:12.725395 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.723805    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:51:12.863781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.863698    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f9893fc-b75a-4bed-925f-c02168ffc76e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f9893fc-b75a-4bed-925f-c02168ffc76e-plugins-config-volume\") pod \"9f9893fc-b75a-4bed-925f-c02168ffc76e\" (UID: \"9f9893fc-b75a-4bed-925f-c02168ffc76e\") "
Oct 31 00:51:12.863964 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.863867    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f9893fc-b75a-4bed-925f-c02168ffc76e-kube-api-access-wzrz7\" (UniqueName: \"kubernetes.io/projected/9f9893fc-b75a-4bed-925f-c02168ffc76e-kube-api-access-wzrz7\") pod \"9f9893fc-b75a-4bed-925f-c02168ffc76e\" (UID: \"9f9893fc-b75a-4bed-925f-c02168ffc76e\") "
Oct 31 00:51:12.865210 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.864599    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f9893fc-b75a-4bed-925f-c02168ffc76e-plugins-config-volume" pod "9f9893fc-b75a-4bed-925f-c02168ffc76e" (UID: "9f9893fc-b75a-4bed-925f-c02168ffc76e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:51:12.882074 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.882029    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f9893fc-b75a-4bed-925f-c02168ffc76e-kube-api-access-wzrz7" pod "9f9893fc-b75a-4bed-925f-c02168ffc76e" (UID: "9f9893fc-b75a-4bed-925f-c02168ffc76e"). InnerVolumeSpecName "kube-api-access-wzrz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:51:12.888625 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.888602    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm"
Oct 31 00:51:12.965452 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.965416    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19b21c31-d906-4652-81e1-765d08fc3e5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19b21c31-d906-4652-81e1-765d08fc3e5e-plugins-config-volume\") pod \"19b21c31-d906-4652-81e1-765d08fc3e5e\" (UID: \"19b21c31-d906-4652-81e1-765d08fc3e5e\") "
Oct 31 00:51:12.965784 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.965760    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b21c31-d906-4652-81e1-765d08fc3e5e-kube-api-access-p5vxg\" (UniqueName: \"kubernetes.io/projected/19b21c31-d906-4652-81e1-765d08fc3e5e-kube-api-access-p5vxg\") pod \"19b21c31-d906-4652-81e1-765d08fc3e5e\" (UID: \"19b21c31-d906-4652-81e1-765d08fc3e5e\") "
Oct 31 00:51:12.966504 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.966463    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzrz7\" (UniqueName: \"kubernetes.io/projected/9f9893fc-b75a-4bed-925f-c02168ffc76e-kube-api-access-wzrz7\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:51:12.966504 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.966495    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f9893fc-b75a-4bed-925f-c02168ffc76e-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:51:12.966783 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.966656    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b21c31-d906-4652-81e1-765d08fc3e5e-plugins-config-volume" pod "19b21c31-d906-4652-81e1-765d08fc3e5e" (UID: "19b21c31-d906-4652-81e1-765d08fc3e5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:51:12.969135 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:12.969107    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b21c31-d906-4652-81e1-765d08fc3e5e-kube-api-access-p5vxg" pod "19b21c31-d906-4652-81e1-765d08fc3e5e" (UID: "19b21c31-d906-4652-81e1-765d08fc3e5e"). InnerVolumeSpecName "kube-api-access-p5vxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:51:13.068954 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:13.068921    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5vxg\" (UniqueName: \"kubernetes.io/projected/19b21c31-d906-4652-81e1-765d08fc3e5e-kube-api-access-p5vxg\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:51:13.068954 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:13.068951    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19b21c31-d906-4652-81e1-765d08fc3e5e-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:51:13.270385 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:13.270291    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:13.277034 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:13.277007    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:13.583964 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:13.583879    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm"
Oct 31 00:51:13.584143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:13.583884    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm" event={"ID":"19b21c31-d906-4652-81e1-765d08fc3e5e","Type":"ContainerDied","Data":"bc5507cb7e15fc048c497a455024bb145a80d5ce3ca67b86c3b2dea72d3f6980"}
Oct 31 00:51:13.584143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:13.584018    4327 scope.go:122] "RemoveContainer" containerID="5f85f06de98a67b708f185f3df346b475c9242e2ff724c3dd41a20ca4982b9ef"
Oct 31 00:51:13.585548 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:13.585402    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq" event={"ID":"9f9893fc-b75a-4bed-925f-c02168ffc76e","Type":"ContainerDied","Data":"a2df8bde484d53edd68822215944296713033ae5e3eb01a77c23829d038281dd"}
Oct 31 00:51:13.585548 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:13.585408    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq"
Oct 31 00:51:13.587415 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:13.587389    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt" event={"ID":"46ed7530-493b-48d4-ac03-a4d9179504ff","Type":"ContainerStarted","Data":"aa239a74c88519d1ce63bd77b96863351fced16f790a1140af92e1b40d72ea9f"}
Oct 31 00:51:13.587695 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:13.587655    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:13.587838 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:13.587755    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt" event={"ID":"46ed7530-493b-48d4-ac03-a4d9179504ff","Type":"ContainerStarted","Data":"96217f6a42053d3f979e847414deb9bf1d66776677fb4ea114ee62886db2b747"}
Oct 31 00:51:13.588817 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:13.588788    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:51:13.602501 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:13.602351    4327 scope.go:122] "RemoveContainer" containerID="ead9d4cbb81ca58e22d70c3854a6714287e3d59d5ae07275a99ea41853dad039"
Oct 31 00:51:13.655759 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:13.655707    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt" podStartSLOduration=2.6556742399999997 podStartE2EDuration="2.65567424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:13.652005259 +0000 UTC m=+1933.874040801" watchObservedRunningTime="2026-10-31 00:51:13.65567424 +0000 UTC m=+1933.877709784"
Oct 31 00:51:13.695802 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:13.695752    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm"]
Oct 31 00:51:13.699532 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:13.699503    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fa2c15c1-5468c658dc-s9bnm"]
Oct 31 00:51:13.737571 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:13.737538    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq"]
Oct 31 00:51:13.741182 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:13.741155    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fa2c15c1-5f4fcb878c-qrppq"]
Oct 31 00:51:13.931398 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:13.931366    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19b21c31-d906-4652-81e1-765d08fc3e5e" path="/var/lib/kubelet/pods/19b21c31-d906-4652-81e1-765d08fc3e5e/volumes"
Oct 31 00:51:13.931973 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:13.931955    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f9893fc-b75a-4bed-925f-c02168ffc76e" path="/var/lib/kubelet/pods/9f9893fc-b75a-4bed-925f-c02168ffc76e/volumes"
Oct 31 00:51:14.099937 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:14.099907    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:14.106275 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:14.106250    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:14.597555 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:14.597519    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:14.598458 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:14.598439    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:51:15.294554 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:51:15.294517    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:51:17.409191 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.409153    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz"]
Oct 31 00:51:17.410252 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.410231    4327 cpu_manager.go:414] "containerMap: removing container" podUID="19b21c31-d906-4652-81e1-765d08fc3e5e" containerName="payload-pre-processing"
Oct 31 00:51:17.410252 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.410250    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b21c31-d906-4652-81e1-765d08fc3e5e" containerName="payload-pre-processing"
Oct 31 00:51:17.410440 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.410286    4327 cpu_manager.go:414] "containerMap: removing container" podUID="9f9893fc-b75a-4bed-925f-c02168ffc76e" containerName="payload-processing"
Oct 31 00:51:17.410440 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.410292    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f9893fc-b75a-4bed-925f-c02168ffc76e" containerName="payload-processing"
Oct 31 00:51:17.410440 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.410424    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f9893fc-b75a-4bed-925f-c02168ffc76e" containerName="payload-processing"
Oct 31 00:51:17.410440 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.410440    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b21c31-d906-4652-81e1-765d08fc3e5e" containerName="payload-pre-processing"
Oct 31 00:51:17.416322 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.416299    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz"
Oct 31 00:51:17.451087 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.451029    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4d5621-dockercfg-8d7zd\""
Oct 31 00:51:17.451317 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.451177    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4d5621\""
Oct 31 00:51:17.567702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.560000    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz"]
Oct 31 00:51:17.592703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.584876    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dccc262c-1c89-428b-a7a0-5d931327bf82-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz\" (UID: \"dccc262c-1c89-428b-a7a0-5d931327bf82\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz"
Oct 31 00:51:17.602705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.599901    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg5sj\" (UniqueName: \"kubernetes.io/projected/dccc262c-1c89-428b-a7a0-5d931327bf82-kube-api-access-fg5sj\") pod \"payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz\" (UID: \"dccc262c-1c89-428b-a7a0-5d931327bf82\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz"
Oct 31 00:51:17.688767 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.687755    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs"]
Oct 31 00:51:17.725754 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.725715    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dccc262c-1c89-428b-a7a0-5d931327bf82-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz\" (UID: \"dccc262c-1c89-428b-a7a0-5d931327bf82\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz"
Oct 31 00:51:17.725980 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.725910    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg5sj\" (UniqueName: \"kubernetes.io/projected/dccc262c-1c89-428b-a7a0-5d931327bf82-kube-api-access-fg5sj\") pod \"payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz\" (UID: \"dccc262c-1c89-428b-a7a0-5d931327bf82\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz"
Oct 31 00:51:17.727339 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.727310    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dccc262c-1c89-428b-a7a0-5d931327bf82-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz\" (UID: \"dccc262c-1c89-428b-a7a0-5d931327bf82\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz"
Oct 31 00:51:17.762705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.760014    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs"]
Oct 31 00:51:17.762705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.760140    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs"
Oct 31 00:51:17.797873 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.797838    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg5sj\" (UniqueName: \"kubernetes.io/projected/dccc262c-1c89-428b-a7a0-5d931327bf82-kube-api-access-fg5sj\") pod \"payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz\" (UID: \"dccc262c-1c89-428b-a7a0-5d931327bf82\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz"
Oct 31 00:51:17.830757 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.827186    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9369d90-6579-4f6a-befa-eb8a84e09bc0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs\" (UID: \"a9369d90-6579-4f6a-befa-eb8a84e09bc0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs"
Oct 31 00:51:17.830757 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.827409    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pwrt\" (UniqueName: \"kubernetes.io/projected/a9369d90-6579-4f6a-befa-eb8a84e09bc0-kube-api-access-4pwrt\") pod \"payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs\" (UID: \"a9369d90-6579-4f6a-befa-eb8a84e09bc0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs"
Oct 31 00:51:17.925993 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.925961    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz"
Oct 31 00:51:17.928750 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.928717    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pwrt\" (UniqueName: \"kubernetes.io/projected/a9369d90-6579-4f6a-befa-eb8a84e09bc0-kube-api-access-4pwrt\") pod \"payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs\" (UID: \"a9369d90-6579-4f6a-befa-eb8a84e09bc0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs"
Oct 31 00:51:17.928883 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.928823    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9369d90-6579-4f6a-befa-eb8a84e09bc0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs\" (UID: \"a9369d90-6579-4f6a-befa-eb8a84e09bc0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs"
Oct 31 00:51:17.929613 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.929562    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9369d90-6579-4f6a-befa-eb8a84e09bc0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs\" (UID: \"a9369d90-6579-4f6a-befa-eb8a84e09bc0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs"
Oct 31 00:51:17.953341 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:17.953313    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pwrt\" (UniqueName: \"kubernetes.io/projected/a9369d90-6579-4f6a-befa-eb8a84e09bc0-kube-api-access-4pwrt\") pod \"payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs\" (UID: \"a9369d90-6579-4f6a-befa-eb8a84e09bc0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs"
Oct 31 00:51:18.119583 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:18.119553    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs"
Oct 31 00:51:18.235735 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:18.235700    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz"]
Oct 31 00:51:18.237085 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:51:18.237050    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddccc262c_1c89_428b_a7a0_5d931327bf82.slice/crio-5b97e85a12ea4646d32dd787bcad0134025f074ec0ac20abaae10394f3fd4976 WatchSource:0}: Error finding container 5b97e85a12ea4646d32dd787bcad0134025f074ec0ac20abaae10394f3fd4976: Status 404 returned error can't find the container with id 5b97e85a12ea4646d32dd787bcad0134025f074ec0ac20abaae10394f3fd4976
Oct 31 00:51:18.443088 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:51:18.443046    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9369d90_6579_4f6a_befa_eb8a84e09bc0.slice/crio-83a0fe4c92a8d1f68033330cbe0a8708c565e51603e16d153b0bb8b83961babf WatchSource:0}: Error finding container 83a0fe4c92a8d1f68033330cbe0a8708c565e51603e16d153b0bb8b83961babf: Status 404 returned error can't find the container with id 83a0fe4c92a8d1f68033330cbe0a8708c565e51603e16d153b0bb8b83961babf
Oct 31 00:51:18.443443 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:18.443342    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs"]
Oct 31 00:51:18.810344 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:18.810244    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz" event={"ID":"dccc262c-1c89-428b-a7a0-5d931327bf82","Type":"ContainerStarted","Data":"501f86f6c351e8129eb3ccaa001b5d72491fd4cb9ae2c59ba5f0a99233cc3406"}
Oct 31 00:51:18.810344 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:18.810313    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz" event={"ID":"dccc262c-1c89-428b-a7a0-5d931327bf82","Type":"ContainerStarted","Data":"5b97e85a12ea4646d32dd787bcad0134025f074ec0ac20abaae10394f3fd4976"}
Oct 31 00:51:18.810344 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:18.810343    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz"
Oct 31 00:51:18.812042 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:18.812012    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs" event={"ID":"a9369d90-6579-4f6a-befa-eb8a84e09bc0","Type":"ContainerStarted","Data":"96f71e10b231416b3a68ce099d40f41e8eb80842a419478e38da90a444c87566"}
Oct 31 00:51:18.812159 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:18.812062    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs" event={"ID":"a9369d90-6579-4f6a-befa-eb8a84e09bc0","Type":"ContainerStarted","Data":"83a0fe4c92a8d1f68033330cbe0a8708c565e51603e16d153b0bb8b83961babf"}
Oct 31 00:51:18.812159 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:18.812100    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs"
Oct 31 00:51:18.857261 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:18.857183    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz" podStartSLOduration=1.857149532 podStartE2EDuration="1.857149532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:18.851386003 +0000 UTC m=+1939.073421545" watchObservedRunningTime="2026-10-31 00:51:18.857149532 +0000 UTC m=+1939.079185075"
Oct 31 00:51:18.896913 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:18.896864    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs" podStartSLOduration=1.8968516260000001 podStartE2EDuration="1.896851626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:18.894019484 +0000 UTC m=+1939.116055046" watchObservedRunningTime="2026-10-31 00:51:18.896851626 +0000 UTC m=+1939.118887167"
Oct 31 00:51:19.451533 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:19.451497    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6"]
Oct 31 00:51:19.458344 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:19.458320    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6"
Oct 31 00:51:19.461784 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:19.461744    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-262tw\" (UniqueName: \"kubernetes.io/projected/147e9302-09cd-4e2c-8a01-e31f299de460-kube-api-access-262tw\") pod \"payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6\" (UID: \"147e9302-09cd-4e2c-8a01-e31f299de460\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6"
Oct 31 00:51:19.461996 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:19.461975    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147e9302-09cd-4e2c-8a01-e31f299de460-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6\" (UID: \"147e9302-09cd-4e2c-8a01-e31f299de460\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6"
Oct 31 00:51:19.495156 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:19.495103    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6"]
Oct 31 00:51:19.499750 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:19.499716    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bbb902-65d446d445-znrq5"]
Oct 31 00:51:19.499897 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:19.499822    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-bbb902\""
Oct 31 00:51:19.502806 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:19.502785    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-bbb902-dockercfg-btcbm\""
Oct 31 00:51:19.505466 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:19.505448    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bbb902-65d446d445-znrq5"
Oct 31 00:51:19.533336 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:19.533304    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bbb902-65d446d445-znrq5"]
Oct 31 00:51:19.603701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:19.600320    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-262tw\" (UniqueName: \"kubernetes.io/projected/147e9302-09cd-4e2c-8a01-e31f299de460-kube-api-access-262tw\") pod \"payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6\" (UID: \"147e9302-09cd-4e2c-8a01-e31f299de460\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6"
Oct 31 00:51:19.603701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:19.600407    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59774227-9989-44ea-9e0d-ba76cd9e544d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bbb902-65d446d445-znrq5\" (UID: \"59774227-9989-44ea-9e0d-ba76cd9e544d\") " pod="openshift-ingress/payload-processing-e2e-api-a-bbb902-65d446d445-znrq5"
Oct 31 00:51:19.603701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:19.600530    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfn89\" (UniqueName: \"kubernetes.io/projected/59774227-9989-44ea-9e0d-ba76cd9e544d-kube-api-access-sfn89\") pod \"payload-processing-e2e-api-a-bbb902-65d446d445-znrq5\" (UID: \"59774227-9989-44ea-9e0d-ba76cd9e544d\") " pod="openshift-ingress/payload-processing-e2e-api-a-bbb902-65d446d445-znrq5"
Oct 31 00:51:19.603701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:19.600600    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147e9302-09cd-4e2c-8a01-e31f299de460-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6\" (UID: \"147e9302-09cd-4e2c-8a01-e31f299de460\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6"
Oct 31 00:51:19.603701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:19.601547    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147e9302-09cd-4e2c-8a01-e31f299de460-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6\" (UID: \"147e9302-09cd-4e2c-8a01-e31f299de460\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6"
Oct 31 00:51:19.703750 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:19.701700    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfn89\" (UniqueName: \"kubernetes.io/projected/59774227-9989-44ea-9e0d-ba76cd9e544d-kube-api-access-sfn89\") pod \"payload-processing-e2e-api-a-bbb902-65d446d445-znrq5\" (UID: \"59774227-9989-44ea-9e0d-ba76cd9e544d\") " pod="openshift-ingress/payload-processing-e2e-api-a-bbb902-65d446d445-znrq5"
Oct 31 00:51:19.721704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:19.713055    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59774227-9989-44ea-9e0d-ba76cd9e544d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bbb902-65d446d445-znrq5\" (UID: \"59774227-9989-44ea-9e0d-ba76cd9e544d\") " pod="openshift-ingress/payload-processing-e2e-api-a-bbb902-65d446d445-znrq5"
Oct 31 00:51:19.721704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:19.714086    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59774227-9989-44ea-9e0d-ba76cd9e544d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bbb902-65d446d445-znrq5\" (UID: \"59774227-9989-44ea-9e0d-ba76cd9e544d\") " pod="openshift-ingress/payload-processing-e2e-api-a-bbb902-65d446d445-znrq5"
Oct 31 00:51:19.836701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:19.836610    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-262tw\" (UniqueName: \"kubernetes.io/projected/147e9302-09cd-4e2c-8a01-e31f299de460-kube-api-access-262tw\") pod \"payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6\" (UID: \"147e9302-09cd-4e2c-8a01-e31f299de460\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6"
Oct 31 00:51:19.889434 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:19.889371    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfn89\" (UniqueName: \"kubernetes.io/projected/59774227-9989-44ea-9e0d-ba76cd9e544d-kube-api-access-sfn89\") pod \"payload-processing-e2e-api-a-bbb902-65d446d445-znrq5\" (UID: \"59774227-9989-44ea-9e0d-ba76cd9e544d\") " pod="openshift-ingress/payload-processing-e2e-api-a-bbb902-65d446d445-znrq5"
Oct 31 00:51:20.081750 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:20.076558    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6"
Oct 31 00:51:20.121366 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:20.121338    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bbb902-65d446d445-znrq5"
Oct 31 00:51:20.473716 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:20.463635    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6"]
Oct 31 00:51:20.683585 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:20.683554    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bbb902-65d446d445-znrq5"]
Oct 31 00:51:20.989757 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:20.973273    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bbb902-65d446d445-znrq5" event={"ID":"59774227-9989-44ea-9e0d-ba76cd9e544d","Type":"ContainerStarted","Data":"8c6e4fd528118e2430ff55981d77e77a8c2a9293acf8aab0b27be13c0f10315b"}
Oct 31 00:51:20.989757 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:20.982258    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6" event={"ID":"147e9302-09cd-4e2c-8a01-e31f299de460","Type":"ContainerStarted","Data":"07cac6a5aeb3ee7c387c616b7ea8f17654a62d99c293788cbedae6dc4d4ed1ac"}
Oct 31 00:51:20.989757 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:20.982322    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6" event={"ID":"147e9302-09cd-4e2c-8a01-e31f299de460","Type":"ContainerStarted","Data":"726db652ae7b46e8ab3d6285976ba814bc4af4f9ee2ab9b71779b543e59c2721"}
Oct 31 00:51:20.989757 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:20.983746    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6"
Oct 31 00:51:21.091630 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.089814    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"]
Oct 31 00:51:21.321023 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.249436    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6" podStartSLOduration=2.249412641 podStartE2EDuration="2.249412641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:21.053397845 +0000 UTC m=+1941.275433388" watchObservedRunningTime="2026-10-31 00:51:21.249412641 +0000 UTC m=+1941.471448185"
Oct 31 00:51:21.346705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.333901    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"]
Oct 31 00:51:21.361705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.354757    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.379705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.370906    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fd1dfc-openshift-default-dockercfg-tn44b\""
Oct 31 00:51:21.583793 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.583669    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kttf5\" (UniqueName: \"kubernetes.io/projected/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-kube-api-access-kttf5\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.583793 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.583730    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istiod-ca-cert\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.584310 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.583889    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-token\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.584310 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.583959    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-workload-certs\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.584310 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.583988    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-workload-socket\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.584310 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.584077    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-podinfo\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.584310 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.584138    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-data\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.584310 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.584232    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-credential-socket\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.584512 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.584343    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-envoy\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.688249 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.688198    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-token\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.693304 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.693250    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-workload-certs\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.693590 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.693562    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-workload-socket\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.693795 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.693656    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-podinfo\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.693795 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.693730    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-data\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.693928 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.693819    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-workload-certs\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.693928 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.693835    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-credential-socket\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.694110 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.694087    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-envoy\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.694186 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.694134    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-credential-socket\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.694250 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.694203    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kttf5\" (UniqueName: \"kubernetes.io/projected/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-kube-api-access-kttf5\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.694250 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.694228    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istiod-ca-cert\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.694832 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.694807    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-workload-socket\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.694959 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.694837    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-data\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.695123 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.695100    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istiod-ca-cert\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.704952 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.702662    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-envoy\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.726364 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.726328    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-podinfo\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.744722 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.744339    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-token\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.785406 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.783838    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kttf5\" (UniqueName: \"kubernetes.io/projected/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-kube-api-access-kttf5\") pod \"e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") " pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.785406 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.784012    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:21.991899 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.991849    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bbb902-65d446d445-znrq5" event={"ID":"59774227-9989-44ea-9e0d-ba76cd9e544d","Type":"ContainerStarted","Data":"47bbe56a0b81774cca7270f01b7659e15f47b8be86b9f301f1e708d71379aefd"}
Oct 31 00:51:21.992103 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:21.991929    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-bbb902-65d446d445-znrq5"
Oct 31 00:51:22.045003 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:22.044941    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-bbb902-65d446d445-znrq5" podStartSLOduration=3.044922771 podStartE2EDuration="3.044922771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:22.04217153 +0000 UTC m=+1942.264207073" watchObservedRunningTime="2026-10-31 00:51:22.044922771 +0000 UTC m=+1942.266958313"
Oct 31 00:51:22.196703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:22.180475    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"]
Oct 31 00:51:22.233076 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:51:22.232993    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47727a40_911f_4d8d_8fd6_f19ada8e4ea0.slice/crio-72a457f6907b123accf13fe1477d365c8dccf615cd4dd40dfe058ecd6e1e59cd WatchSource:0}: Error finding container 72a457f6907b123accf13fe1477d365c8dccf615cd4dd40dfe058ecd6e1e59cd: Status 404 returned error can't find the container with id 72a457f6907b123accf13fe1477d365c8dccf615cd4dd40dfe058ecd6e1e59cd
Oct 31 00:51:22.253513 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:22.253464    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:51:22.253830 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:22.253792    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:51:22.254078 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:22.254015    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:51:23.002538 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:23.002499    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv" event={"ID":"47727a40-911f-4d8d-8fd6-f19ada8e4ea0","Type":"ContainerStarted","Data":"8514da30c40f9a548d39bd46eb762a911457d8b78e6f4b792b4a77dfa98caa47"}
Oct 31 00:51:23.003402 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:23.002560    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv" event={"ID":"47727a40-911f-4d8d-8fd6-f19ada8e4ea0","Type":"ContainerStarted","Data":"72a457f6907b123accf13fe1477d365c8dccf615cd4dd40dfe058ecd6e1e59cd"}
Oct 31 00:51:23.075740 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:23.075654    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv" podStartSLOduration=3.075639948 podStartE2EDuration="3.075639948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:23.072503517 +0000 UTC m=+1943.294539060" watchObservedRunningTime="2026-10-31 00:51:23.075639948 +0000 UTC m=+1943.297675489"
Oct 31 00:51:23.785242 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:23.785208    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:23.791797 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:23.791770    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:24.011309 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:24.011272    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:24.012000 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:24.011973    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:51:27.469605 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:27.469568    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28"]
Oct 31 00:51:27.474838 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:27.474815    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28"
Oct 31 00:51:27.495271 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:27.495151    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28"]
Oct 31 00:51:27.501286 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:27.501157    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-fd1dfc-dockercfg-2hrvn\""
Oct 31 00:51:27.501286 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:27.501229    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-fd1dfc\""
Oct 31 00:51:27.509772 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:27.509740    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f113a57a-325b-4508-ad0f-3e9677b3f95d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28\" (UID: \"f113a57a-325b-4508-ad0f-3e9677b3f95d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28"
Oct 31 00:51:27.509995 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:27.509937    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8pbk\" (UniqueName: \"kubernetes.io/projected/f113a57a-325b-4508-ad0f-3e9677b3f95d-kube-api-access-q8pbk\") pod \"payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28\" (UID: \"f113a57a-325b-4508-ad0f-3e9677b3f95d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28"
Oct 31 00:51:27.614796 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:27.614757    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f113a57a-325b-4508-ad0f-3e9677b3f95d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28\" (UID: \"f113a57a-325b-4508-ad0f-3e9677b3f95d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28"
Oct 31 00:51:27.632164 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:27.632127    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8pbk\" (UniqueName: \"kubernetes.io/projected/f113a57a-325b-4508-ad0f-3e9677b3f95d-kube-api-access-q8pbk\") pod \"payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28\" (UID: \"f113a57a-325b-4508-ad0f-3e9677b3f95d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28"
Oct 31 00:51:27.728753 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:27.721558    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f113a57a-325b-4508-ad0f-3e9677b3f95d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28\" (UID: \"f113a57a-325b-4508-ad0f-3e9677b3f95d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28"
Oct 31 00:51:27.797581 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:27.772750    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc"]
Oct 31 00:51:27.868663 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:27.868618    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc"
Oct 31 00:51:27.887636 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:27.887580    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc"]
Oct 31 00:51:27.889746 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:27.889663    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8pbk\" (UniqueName: \"kubernetes.io/projected/f113a57a-325b-4508-ad0f-3e9677b3f95d-kube-api-access-q8pbk\") pod \"payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28\" (UID: \"f113a57a-325b-4508-ad0f-3e9677b3f95d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28"
Oct 31 00:51:27.991989 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:27.991891    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841015e0-30ae-4810-a913-598be430b8fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc\" (UID: \"841015e0-30ae-4810-a913-598be430b8fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc"
Oct 31 00:51:27.992255 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:27.992224    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdp7w\" (UniqueName: \"kubernetes.io/projected/841015e0-30ae-4810-a913-598be430b8fd-kube-api-access-tdp7w\") pod \"payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc\" (UID: \"841015e0-30ae-4810-a913-598be430b8fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc"
Oct 31 00:51:28.094332 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:28.094299    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdp7w\" (UniqueName: \"kubernetes.io/projected/841015e0-30ae-4810-a913-598be430b8fd-kube-api-access-tdp7w\") pod \"payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc\" (UID: \"841015e0-30ae-4810-a913-598be430b8fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc"
Oct 31 00:51:28.094525 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:28.094443    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841015e0-30ae-4810-a913-598be430b8fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc\" (UID: \"841015e0-30ae-4810-a913-598be430b8fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc"
Oct 31 00:51:28.095400 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:28.095367    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841015e0-30ae-4810-a913-598be430b8fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc\" (UID: \"841015e0-30ae-4810-a913-598be430b8fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc"
Oct 31 00:51:28.096605 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:28.096579    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28"
Oct 31 00:51:28.118363 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:28.118322    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdp7w\" (UniqueName: \"kubernetes.io/projected/841015e0-30ae-4810-a913-598be430b8fd-kube-api-access-tdp7w\") pod \"payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc\" (UID: \"841015e0-30ae-4810-a913-598be430b8fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc"
Oct 31 00:51:28.211359 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:28.211319    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc"
Oct 31 00:51:28.409325 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:51:28.408814    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf113a57a_325b_4508_ad0f_3e9677b3f95d.slice/crio-fc449ea5a7f3d14c49262525a0fc4047c1c6241268f25cc509b9ee3aecf88591 WatchSource:0}: Error finding container fc449ea5a7f3d14c49262525a0fc4047c1c6241268f25cc509b9ee3aecf88591: Status 404 returned error can't find the container with id fc449ea5a7f3d14c49262525a0fc4047c1c6241268f25cc509b9ee3aecf88591
Oct 31 00:51:28.414832 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:28.414786    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28"]
Oct 31 00:51:28.531826 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:51:28.531756    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841015e0_30ae_4810_a913_598be430b8fd.slice/crio-a77450732a1516b64afeee1fbea07dbce84c3c3229866266f3cfb3ce45880c9c WatchSource:0}: Error finding container a77450732a1516b64afeee1fbea07dbce84c3c3229866266f3cfb3ce45880c9c: Status 404 returned error can't find the container with id a77450732a1516b64afeee1fbea07dbce84c3c3229866266f3cfb3ce45880c9c
Oct 31 00:51:28.537564 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:28.537530    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc"]
Oct 31 00:51:29.184074 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.184028    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28" event={"ID":"f113a57a-325b-4508-ad0f-3e9677b3f95d","Type":"ContainerStarted","Data":"d2b98a8a41303c17b12e9bdfb78a79c9bb640cbf7d68c8354e1dd9c32104d5b1"}
Oct 31 00:51:29.184285 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.184092    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28" event={"ID":"f113a57a-325b-4508-ad0f-3e9677b3f95d","Type":"ContainerStarted","Data":"fc449ea5a7f3d14c49262525a0fc4047c1c6241268f25cc509b9ee3aecf88591"}
Oct 31 00:51:29.184285 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.184124    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28"
Oct 31 00:51:29.186129 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.186090    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc" event={"ID":"841015e0-30ae-4810-a913-598be430b8fd","Type":"ContainerStarted","Data":"a1f7c1812ca2d57653d4c723de70bff882d8f7d43871abd93487ddd144887b20"}
Oct 31 00:51:29.186258 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.186149    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc" event={"ID":"841015e0-30ae-4810-a913-598be430b8fd","Type":"ContainerStarted","Data":"a77450732a1516b64afeee1fbea07dbce84c3c3229866266f3cfb3ce45880c9c"}
Oct 31 00:51:29.186258 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.186190    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc"
Oct 31 00:51:29.257704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.244515    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28" podStartSLOduration=2.244491568 podStartE2EDuration="2.244491568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:29.242059828 +0000 UTC m=+1949.464095371" watchObservedRunningTime="2026-10-31 00:51:29.244491568 +0000 UTC m=+1949.466527174"
Oct 31 00:51:29.299708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.298177    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc" podStartSLOduration=2.298155594 podStartE2EDuration="2.298155594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:29.286229652 +0000 UTC m=+1949.508265197" watchObservedRunningTime="2026-10-31 00:51:29.298155594 +0000 UTC m=+1949.520191137"
Oct 31 00:51:29.491801 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.474436    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"]
Oct 31 00:51:29.715935 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.710493    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.729168 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.728656    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-208e9f32-openshift-default-dockercfg-rs5gs\""
Oct 31 00:51:29.735351 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.735324    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"]
Oct 31 00:51:29.806191 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.806088    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-credential-socket\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.806191 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.806144    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-workload-certs\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.806383 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.806253    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ff44805-0009-41dc-9415-cc05817837ea-istiod-ca-cert\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.806423 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.806380    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgmlv\" (UniqueName: \"kubernetes.io/projected/5ff44805-0009-41dc-9415-cc05817837ea-kube-api-access-xgmlv\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.806657 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.806632    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-istio-data\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.806731 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.806695    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-istio-envoy\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.806895 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.806872    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-workload-socket\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.806998 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.806977    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ff44805-0009-41dc-9415-cc05817837ea-istio-token\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.807059 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.807040    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ff44805-0009-41dc-9415-cc05817837ea-istio-podinfo\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.931939 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.927156    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-workload-socket\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.931939 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.930861    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ff44805-0009-41dc-9415-cc05817837ea-istio-token\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.931939 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.930900    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ff44805-0009-41dc-9415-cc05817837ea-istio-podinfo\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.931939 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.930975    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-credential-socket\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.931939 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.931002    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-workload-certs\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.931939 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.931035    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ff44805-0009-41dc-9415-cc05817837ea-istiod-ca-cert\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.931939 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.931123    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgmlv\" (UniqueName: \"kubernetes.io/projected/5ff44805-0009-41dc-9415-cc05817837ea-kube-api-access-xgmlv\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.931939 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.931345    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-istio-data\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.931939 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.931380    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-istio-envoy\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.935581 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.933595    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-credential-socket\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.935581 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.934655    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ff44805-0009-41dc-9415-cc05817837ea-istiod-ca-cert\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.935581 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.934815    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-istio-envoy\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.935581 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.935009    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-workload-certs\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.935581 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.935275    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-istio-data\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.936232 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.936193    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-workload-socket\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.944727 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.944700    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs"
Oct 31 00:51:29.944865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.944781    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz"
Oct 31 00:51:29.950493 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.950469    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ff44805-0009-41dc-9415-cc05817837ea-istio-podinfo\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.959474 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.959448    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ff44805-0009-41dc-9415-cc05817837ea-istio-token\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:29.959581 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:29.959471    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgmlv\" (UniqueName: \"kubernetes.io/projected/5ff44805-0009-41dc-9415-cc05817837ea-kube-api-access-xgmlv\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-cjpms\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:30.040967 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:30.040920    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:30.409842 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:30.409811    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"]
Oct 31 00:51:30.415222 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:30.415179    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:51:30.415318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:30.415268    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:51:30.415318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:30.415307    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:51:31.252728 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:31.252654    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms" event={"ID":"5ff44805-0009-41dc-9415-cc05817837ea","Type":"ContainerStarted","Data":"702a622173a4630c5bd54e3e19145cf6f1601c7bef627fed08d39d89452fdaa5"}
Oct 31 00:51:31.253151 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:31.252762    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms" event={"ID":"5ff44805-0009-41dc-9415-cc05817837ea","Type":"ContainerStarted","Data":"b7c73fe105a67b44c49846f73e900702a010cd72c429d8f4df73c38372b1ce03"}
Oct 31 00:51:31.308122 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:31.308056    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms" podStartSLOduration=2.308036329 podStartE2EDuration="2.308036329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:31.30446296 +0000 UTC m=+1951.526498525" watchObservedRunningTime="2026-10-31 00:51:31.308036329 +0000 UTC m=+1951.530071875"
Oct 31 00:51:32.149961 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:32.149097    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:32.175599 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:32.169815    4327 patch_prober.go:29] interesting pod/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 31 00:51:32.175599 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:32.169888    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms" podUID="5ff44805-0009-41dc-9415-cc05817837ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 31 00:51:33.006562 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:33.006483    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-bbb902-65d446d445-znrq5"
Oct 31 00:51:33.006562 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:33.006559    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6"
Oct 31 00:51:33.042072 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:33.042040    4327 patch_prober.go:29] interesting pod/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 31 00:51:33.042288 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:33.042101    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms" podUID="5ff44805-0009-41dc-9415-cc05817837ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 31 00:51:33.456575 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:33.456544    4327 generic.go:356] "Generic (PLEG): container finished" podID="400fc179-6148-4af6-b069-88f843600416" containerID="3504921cf5a6184bc5b78a5f156da93de8034956feeb12fd05501d0a3a920905" exitCode=0
Oct 31 00:51:33.456797 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:33.456613    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv" event={"ID":"400fc179-6148-4af6-b069-88f843600416","Type":"ContainerDied","Data":"3504921cf5a6184bc5b78a5f156da93de8034956feeb12fd05501d0a3a920905"}
Oct 31 00:51:33.457133 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:33.457115    4327 scope.go:122] "RemoveContainer" containerID="3504921cf5a6184bc5b78a5f156da93de8034956feeb12fd05501d0a3a920905"
Oct 31 00:51:33.459201 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:33.459164    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:51:33.459300 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:33.459257    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:51:33.459338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:33.459301    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:51:34.056209 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:34.054306    4327 patch_prober.go:29] interesting pod/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 31 00:51:34.056209 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:34.054375    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms" podUID="5ff44805-0009-41dc-9415-cc05817837ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 31 00:51:34.521971 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:34.520603    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv" event={"ID":"400fc179-6148-4af6-b069-88f843600416","Type":"ContainerStarted","Data":"71d469b3c678d63ecb92fe3295e938187b187d4a1be0b0ba562a64b90f03e5f8"}
Oct 31 00:51:35.043761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:35.042705    4327 patch_prober.go:29] interesting pod/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 31 00:51:35.043761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:35.042786    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms" podUID="5ff44805-0009-41dc-9415-cc05817837ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 31 00:51:35.438108 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:35.438071    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:51:35.438814 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:35.438789    4327 patch_prober.go:29] interesting pod/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 00:51:35.439021 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:35.438992    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv" podUID="400fc179-6148-4af6-b069-88f843600416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 00:51:36.046705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:36.046272    4327 patch_prober.go:29] interesting pod/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 31 00:51:36.046705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:36.046333    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms" podUID="5ff44805-0009-41dc-9415-cc05817837ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 31 00:51:36.434245 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:36.434200    4327 patch_prober.go:29] interesting pod/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 00:51:36.434541 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:36.434503    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv" podUID="400fc179-6148-4af6-b069-88f843600416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 00:51:37.041711 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:37.041602    4327 patch_prober.go:29] interesting pod/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 31 00:51:37.042696 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:37.041665    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms" podUID="5ff44805-0009-41dc-9415-cc05817837ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 31 00:51:37.162251 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:37.162212    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:51:37.442707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:37.436839    4327 patch_prober.go:29] interesting pod/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 00:51:37.442707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:37.436907    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv" podUID="400fc179-6148-4af6-b069-88f843600416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 00:51:38.066171 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:38.066125    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst"]
Oct 31 00:51:38.074177 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:38.074131    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst"
Oct 31 00:51:38.090051 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:38.090019    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-82d887\""
Oct 31 00:51:38.090413 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:38.090390    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-82d887-dockercfg-m2wf8\""
Oct 31 00:51:38.094922 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:38.094847    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst"]
Oct 31 00:51:38.311899 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:38.310919    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gks8z\" (UniqueName: \"kubernetes.io/projected/f17abb15-a890-44cf-a93e-fe9580905cd4-kube-api-access-gks8z\") pod \"payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst\" (UID: \"f17abb15-a890-44cf-a93e-fe9580905cd4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst"
Oct 31 00:51:38.311899 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:38.311400    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f17abb15-a890-44cf-a93e-fe9580905cd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst\" (UID: \"f17abb15-a890-44cf-a93e-fe9580905cd4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst"
Oct 31 00:51:38.401020 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:38.400726    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j"]
Oct 31 00:51:38.642706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:38.495547    4327 patch_prober.go:29] interesting pod/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 00:51:38.642706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:38.576406    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv" podUID="400fc179-6148-4af6-b069-88f843600416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 00:51:38.642706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:38.604133    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f17abb15-a890-44cf-a93e-fe9580905cd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst\" (UID: \"f17abb15-a890-44cf-a93e-fe9580905cd4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst"
Oct 31 00:51:38.642706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:38.604256    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gks8z\" (UniqueName: \"kubernetes.io/projected/f17abb15-a890-44cf-a93e-fe9580905cd4-kube-api-access-gks8z\") pod \"payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst\" (UID: \"f17abb15-a890-44cf-a93e-fe9580905cd4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst"
Oct 31 00:51:38.642706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:38.613381    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f17abb15-a890-44cf-a93e-fe9580905cd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst\" (UID: \"f17abb15-a890-44cf-a93e-fe9580905cd4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst"
Oct 31 00:51:38.915147 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:38.803502    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gks8z\" (UniqueName: \"kubernetes.io/projected/f17abb15-a890-44cf-a93e-fe9580905cd4-kube-api-access-gks8z\") pod \"payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst\" (UID: \"f17abb15-a890-44cf-a93e-fe9580905cd4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst"
Oct 31 00:51:39.020105 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:39.020061    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j"]
Oct 31 00:51:39.020490 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:39.020471    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:39.045608 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:39.027434    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j"
Oct 31 00:51:39.045608 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:39.042037    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:39.058505 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:39.048659    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst"
Oct 31 00:51:39.079674 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:39.079633    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:51:39.196492 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:39.187748    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz" podUID="b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:51:39.295321 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:39.295285    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vclh5\" (UniqueName: \"kubernetes.io/projected/dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e-kube-api-access-vclh5\") pod \"payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j\" (UID: \"dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e\") " pod="openshift-ingress/payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j"
Oct 31 00:51:39.302063 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:39.302024    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j\" (UID: \"dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e\") " pod="openshift-ingress/payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j"
Oct 31 00:51:39.411785 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:39.411542    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vclh5\" (UniqueName: \"kubernetes.io/projected/dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e-kube-api-access-vclh5\") pod \"payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j\" (UID: \"dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e\") " pod="openshift-ingress/payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j"
Oct 31 00:51:39.411785 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:39.411625    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j\" (UID: \"dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e\") " pod="openshift-ingress/payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j"
Oct 31 00:51:39.450019 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:39.448467    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j\" (UID: \"dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e\") " pod="openshift-ingress/payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j"
Oct 31 00:51:39.450019 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:39.449000    4327 patch_prober.go:29] interesting pod/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 00:51:39.450019 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:39.449057    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv" podUID="400fc179-6148-4af6-b069-88f843600416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 00:51:39.473568 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:39.469999    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vclh5\" (UniqueName: \"kubernetes.io/projected/dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e-kube-api-access-vclh5\") pod \"payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j\" (UID: \"dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e\") " pod="openshift-ingress/payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j"
Oct 31 00:51:39.697740 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:39.697277    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j"
Oct 31 00:51:39.762407 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:39.761995    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst"]
Oct 31 00:51:40.000523 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:40.000425    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst" event={"ID":"f17abb15-a890-44cf-a93e-fe9580905cd4","Type":"ContainerStarted","Data":"cdfa86bf177cf3d1081f2dae218f55b0a8671c25ae7e463f6cd440ad4ffa5019"}
Oct 31 00:51:40.000523 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:40.000483    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst" event={"ID":"f17abb15-a890-44cf-a93e-fe9580905cd4","Type":"ContainerStarted","Data":"0da1fd2e177bf32ce1057a2d0c56d9bcbde16487645b1e8ba6ab9438bfb12da8"}
Oct 31 00:51:40.000788 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:40.000519    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst"
Oct 31 00:51:40.056124 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:40.056074    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst" podStartSLOduration=2.05606005 podStartE2EDuration="2.05606005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:40.053220452 +0000 UTC m=+1960.275255994" watchObservedRunningTime="2026-10-31 00:51:40.05606005 +0000 UTC m=+1960.278095591"
Oct 31 00:51:40.099323 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:40.099289    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j"]
Oct 31 00:51:40.099717 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:51:40.099608    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6bec5c_5bf7_4e9c_b33b_caf337aeb76e.slice/crio-f76c2ec98761ac2d04501cff5de49d9743a0799f1d7cd6f2cde429c4e29fe041 WatchSource:0}: Error finding container f76c2ec98761ac2d04501cff5de49d9743a0799f1d7cd6f2cde429c4e29fe041: Status 404 returned error can't find the container with id f76c2ec98761ac2d04501cff5de49d9743a0799f1d7cd6f2cde429c4e29fe041
Oct 31 00:51:40.243702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:40.239921    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc"
Oct 31 00:51:40.345700 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:40.342134    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28"
Oct 31 00:51:40.470747 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:40.466813    4327 patch_prober.go:29] interesting pod/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 00:51:40.470747 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:40.466878    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv" podUID="400fc179-6148-4af6-b069-88f843600416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 00:51:41.009738 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:41.009699    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j" event={"ID":"dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e","Type":"ContainerStarted","Data":"83583a4d2cdcb97f23a0c0345111114a0b538d4962d19029e30949ee64123dae"}
Oct 31 00:51:41.009971 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:41.009764    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j" event={"ID":"dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e","Type":"ContainerStarted","Data":"f76c2ec98761ac2d04501cff5de49d9743a0799f1d7cd6f2cde429c4e29fe041"}
Oct 31 00:51:41.059003 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:41.058936    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j" podStartSLOduration=3.058920279 podStartE2EDuration="3.058920279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:41.058294835 +0000 UTC m=+1961.280330378" watchObservedRunningTime="2026-10-31 00:51:41.058920279 +0000 UTC m=+1961.280955852"
Oct 31 00:51:41.434703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:41.434078    4327 patch_prober.go:29] interesting pod/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 00:51:41.434703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:41.434146    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv" podUID="400fc179-6148-4af6-b069-88f843600416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 00:51:42.022606 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:42.022570    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j"
Oct 31 00:51:42.439169 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:42.439141    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:51:42.440265 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:42.440242    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:51:45.393701 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:51:45.389830    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:51:49.860525 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:49.860486    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2"]
Oct 31 00:51:49.924034 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:49.924000    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2"]
Oct 31 00:51:51.012641 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:51.012611    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst"
Oct 31 00:51:53.035528 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:53.035496    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j"
Oct 31 00:51:53.865662 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:53.865096    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"]
Oct 31 00:51:53.865662 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:53.865426    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh" podUID="aca0da87-ad0a-45cc-a022-d4946e7fab22" containerName="istio-proxy" containerID="cri-o://af2e4eca3107c1082dd2626643c999ae9eb38e7f41b782c99e52982e2e64ad3f" gracePeriod=30
Oct 31 00:51:54.861237 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:54.861162    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2" podUID="d8424735-42e3-406b-897c-9abe18751264" containerName="payload-processing" containerID="cri-o://b8df9e7bb2cf8cb0365b7fa7931a95863795472f1db50ad11532259f45311a93" gracePeriod=25
Oct 31 00:51:54.929293 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:54.929252    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2" podUID="5740efc6-1a2d-4762-93ad-ce5310f03c7a" containerName="payload-pre-processing" containerID="cri-o://ce23b983f8a1be07e3e118480789461c4bc345b4404284a7c7405c2b3609a105" gracePeriod=25
Oct 31 00:51:55.249238 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.249202    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2"
Oct 31 00:51:55.279488 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.279461    4327 generic.go:356] "Generic (PLEG): container finished" podID="d8424735-42e3-406b-897c-9abe18751264" containerID="b8df9e7bb2cf8cb0365b7fa7931a95863795472f1db50ad11532259f45311a93" exitCode=0
Oct 31 00:51:55.279607 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.279515    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2"
Oct 31 00:51:55.279607 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.279512    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2" event={"ID":"d8424735-42e3-406b-897c-9abe18751264","Type":"ContainerDied","Data":"b8df9e7bb2cf8cb0365b7fa7931a95863795472f1db50ad11532259f45311a93"}
Oct 31 00:51:55.279724 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.279643    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2" event={"ID":"d8424735-42e3-406b-897c-9abe18751264","Type":"ContainerDied","Data":"1e935a424da1ec5e02ad8a38c77e212dbc90b88f95298b3e787684f77c92284f"}
Oct 31 00:51:55.279724 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.279671    4327 scope.go:122] "RemoveContainer" containerID="b8df9e7bb2cf8cb0365b7fa7931a95863795472f1db50ad11532259f45311a93"
Oct 31 00:51:55.281339 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.281314    4327 generic.go:356] "Generic (PLEG): container finished" podID="5740efc6-1a2d-4762-93ad-ce5310f03c7a" containerID="ce23b983f8a1be07e3e118480789461c4bc345b4404284a7c7405c2b3609a105" exitCode=0
Oct 31 00:51:55.281458 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.281401    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2" event={"ID":"5740efc6-1a2d-4762-93ad-ce5310f03c7a","Type":"ContainerDied","Data":"ce23b983f8a1be07e3e118480789461c4bc345b4404284a7c7405c2b3609a105"}
Oct 31 00:51:55.281458 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.281440    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2" event={"ID":"5740efc6-1a2d-4762-93ad-ce5310f03c7a","Type":"ContainerDied","Data":"191185dd924bc3df99307b75b8f2e3f15465909f898e1e0fc5344fd0043f1296"}
Oct 31 00:51:55.281561 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.281459    4327 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="191185dd924bc3df99307b75b8f2e3f15465909f898e1e0fc5344fd0043f1296"
Oct 31 00:51:55.290493 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.290474    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2"
Oct 31 00:51:55.307212 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.307188    4327 scope.go:122] "RemoveContainer" containerID="b8df9e7bb2cf8cb0365b7fa7931a95863795472f1db50ad11532259f45311a93"
Oct 31 00:51:55.335763 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:51:55.334291    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8df9e7bb2cf8cb0365b7fa7931a95863795472f1db50ad11532259f45311a93\": container with ID starting with b8df9e7bb2cf8cb0365b7fa7931a95863795472f1db50ad11532259f45311a93 not found: ID does not exist" containerID="b8df9e7bb2cf8cb0365b7fa7931a95863795472f1db50ad11532259f45311a93"
Oct 31 00:51:55.335763 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.334343    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8df9e7bb2cf8cb0365b7fa7931a95863795472f1db50ad11532259f45311a93"} err="failed to get container status \"b8df9e7bb2cf8cb0365b7fa7931a95863795472f1db50ad11532259f45311a93\": rpc error: code = NotFound desc = could not find container \"b8df9e7bb2cf8cb0365b7fa7931a95863795472f1db50ad11532259f45311a93\": container with ID starting with b8df9e7bb2cf8cb0365b7fa7931a95863795472f1db50ad11532259f45311a93 not found: ID does not exist"
Oct 31 00:51:55.346856 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.345357    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8424735-42e3-406b-897c-9abe18751264-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8424735-42e3-406b-897c-9abe18751264-plugins-config-volume\") pod \"d8424735-42e3-406b-897c-9abe18751264\" (UID: \"d8424735-42e3-406b-897c-9abe18751264\") "
Oct 31 00:51:55.346856 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.345577    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8424735-42e3-406b-897c-9abe18751264-kube-api-access-dg2s4\" (UniqueName: \"kubernetes.io/projected/d8424735-42e3-406b-897c-9abe18751264-kube-api-access-dg2s4\") pod \"d8424735-42e3-406b-897c-9abe18751264\" (UID: \"d8424735-42e3-406b-897c-9abe18751264\") "
Oct 31 00:51:55.346856 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.345862    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8424735-42e3-406b-897c-9abe18751264-plugins-config-volume" pod "d8424735-42e3-406b-897c-9abe18751264" (UID: "d8424735-42e3-406b-897c-9abe18751264"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:51:55.346856 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.346347    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8424735-42e3-406b-897c-9abe18751264-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:51:55.352186 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.352157    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8424735-42e3-406b-897c-9abe18751264-kube-api-access-dg2s4" pod "d8424735-42e3-406b-897c-9abe18751264" (UID: "d8424735-42e3-406b-897c-9abe18751264"). InnerVolumeSpecName "kube-api-access-dg2s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:51:55.447810 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.447701    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5740efc6-1a2d-4762-93ad-ce5310f03c7a-kube-api-access-zv6cb\" (UniqueName: \"kubernetes.io/projected/5740efc6-1a2d-4762-93ad-ce5310f03c7a-kube-api-access-zv6cb\") pod \"5740efc6-1a2d-4762-93ad-ce5310f03c7a\" (UID: \"5740efc6-1a2d-4762-93ad-ce5310f03c7a\") "
Oct 31 00:51:55.447810 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.447761    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5740efc6-1a2d-4762-93ad-ce5310f03c7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5740efc6-1a2d-4762-93ad-ce5310f03c7a-plugins-config-volume\") pod \"5740efc6-1a2d-4762-93ad-ce5310f03c7a\" (UID: \"5740efc6-1a2d-4762-93ad-ce5310f03c7a\") "
Oct 31 00:51:55.448222 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.448191    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5740efc6-1a2d-4762-93ad-ce5310f03c7a-plugins-config-volume" pod "5740efc6-1a2d-4762-93ad-ce5310f03c7a" (UID: "5740efc6-1a2d-4762-93ad-ce5310f03c7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:51:55.448496 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.448476    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg2s4\" (UniqueName: \"kubernetes.io/projected/d8424735-42e3-406b-897c-9abe18751264-kube-api-access-dg2s4\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:51:55.448574 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.448498    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5740efc6-1a2d-4762-93ad-ce5310f03c7a-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:51:55.450106 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.450082    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5740efc6-1a2d-4762-93ad-ce5310f03c7a-kube-api-access-zv6cb" pod "5740efc6-1a2d-4762-93ad-ce5310f03c7a" (UID: "5740efc6-1a2d-4762-93ad-ce5310f03c7a"). InnerVolumeSpecName "kube-api-access-zv6cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:51:55.550506 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.550467    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv6cb\" (UniqueName: \"kubernetes.io/projected/5740efc6-1a2d-4762-93ad-ce5310f03c7a-kube-api-access-zv6cb\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:51:55.628242 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.628198    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2"]
Oct 31 00:51:55.640534 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.640496    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3b1945-577dddb78f-trzq2"]
Oct 31 00:51:55.941410 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:55.941374    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8424735-42e3-406b-897c-9abe18751264" path="/var/lib/kubelet/pods/d8424735-42e3-406b-897c-9abe18751264/volumes"
Oct 31 00:51:56.290716 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:56.290627    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2"
Oct 31 00:51:56.346824 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:56.346795    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2"]
Oct 31 00:51:56.358445 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:56.358415    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3b1945-575c89bbf9-hf7c2"]
Oct 31 00:51:57.492762 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:57.492728    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl"]
Oct 31 00:51:57.493536 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:57.493521    4327 cpu_manager.go:414] "containerMap: removing container" podUID="d8424735-42e3-406b-897c-9abe18751264" containerName="payload-processing"
Oct 31 00:51:57.493536 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:57.493536    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8424735-42e3-406b-897c-9abe18751264" containerName="payload-processing"
Oct 31 00:51:57.493613 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:57.493551    4327 cpu_manager.go:414] "containerMap: removing container" podUID="5740efc6-1a2d-4762-93ad-ce5310f03c7a" containerName="payload-pre-processing"
Oct 31 00:51:57.493613 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:57.493557    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5740efc6-1a2d-4762-93ad-ce5310f03c7a" containerName="payload-pre-processing"
Oct 31 00:51:57.493715 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:57.493710    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8424735-42e3-406b-897c-9abe18751264" containerName="payload-processing"
Oct 31 00:51:57.493781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:57.493721    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="5740efc6-1a2d-4762-93ad-ce5310f03c7a" containerName="payload-pre-processing"
Oct 31 00:51:57.499220 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:57.499199    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl"
Oct 31 00:51:57.523894 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:57.523854    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-bbb902-serving-cert\""
Oct 31 00:51:57.531189 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:57.531157    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl"]
Oct 31 00:51:57.591094 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:57.591056    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1783fef-e971-4cce-861a-37b8c232d0b3-openshift-service-ca\") pod \"maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl\" (UID: \"b1783fef-e971-4cce-861a-37b8c232d0b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl"
Oct 31 00:51:57.591284 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:57.591157    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1783fef-e971-4cce-861a-37b8c232d0b3-metrics-certs\") pod \"maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl\" (UID: \"b1783fef-e971-4cce-861a-37b8c232d0b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl"
Oct 31 00:51:57.591451 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:57.591419    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1783fef-e971-4cce-861a-37b8c232d0b3-maas-api-tls\") pod \"maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl\" (UID: \"b1783fef-e971-4cce-861a-37b8c232d0b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl"
Oct 31 00:51:57.591580 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:57.591512    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn695\" (UniqueName: \"kubernetes.io/projected/b1783fef-e971-4cce-861a-37b8c232d0b3-kube-api-access-rn695\") pod \"maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl\" (UID: \"b1783fef-e971-4cce-861a-37b8c232d0b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl"
Oct 31 00:51:57.701789 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:57.692929    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1783fef-e971-4cce-861a-37b8c232d0b3-maas-api-tls\") pod \"maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl\" (UID: \"b1783fef-e971-4cce-861a-37b8c232d0b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl"
Oct 31 00:51:57.701789 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:57.693059    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn695\" (UniqueName: \"kubernetes.io/projected/b1783fef-e971-4cce-861a-37b8c232d0b3-kube-api-access-rn695\") pod \"maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl\" (UID: \"b1783fef-e971-4cce-861a-37b8c232d0b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl"
Oct 31 00:51:57.701789 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:57.693102    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1783fef-e971-4cce-861a-37b8c232d0b3-openshift-service-ca\") pod \"maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl\" (UID: \"b1783fef-e971-4cce-861a-37b8c232d0b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl"
Oct 31 00:51:57.701789 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:57.693163    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1783fef-e971-4cce-861a-37b8c232d0b3-metrics-certs\") pod \"maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl\" (UID: \"b1783fef-e971-4cce-861a-37b8c232d0b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl"
Oct 31 00:51:57.701789 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:51:57.696966    4327 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-serving-cert: secret "maas-api-e2e-api-a-bbb902-serving-cert" not found
Oct 31 00:51:57.701789 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:51:57.697080    4327 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1783fef-e971-4cce-861a-37b8c232d0b3-maas-api-tls podName:b1783fef-e971-4cce-861a-37b8c232d0b3 nodeName:}" failed. No retries permitted until 2026-10-31 00:51:58.197042158 +0000 UTC m=+1978.419077702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b1783fef-e971-4cce-861a-37b8c232d0b3-maas-api-tls") pod "maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl" (UID: "b1783fef-e971-4cce-861a-37b8c232d0b3") : secret "maas-api-e2e-api-a-bbb902-serving-cert" not found
Oct 31 00:51:57.701789 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:57.698149    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1783fef-e971-4cce-861a-37b8c232d0b3-openshift-service-ca\") pod \"maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl\" (UID: \"b1783fef-e971-4cce-861a-37b8c232d0b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl"
Oct 31 00:51:57.707228 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:57.706583    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1783fef-e971-4cce-861a-37b8c232d0b3-metrics-certs\") pod \"maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl\" (UID: \"b1783fef-e971-4cce-861a-37b8c232d0b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl"
Oct 31 00:51:57.733709 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:57.723946    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn695\" (UniqueName: \"kubernetes.io/projected/b1783fef-e971-4cce-861a-37b8c232d0b3-kube-api-access-rn695\") pod \"maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl\" (UID: \"b1783fef-e971-4cce-861a-37b8c232d0b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl"
Oct 31 00:51:57.924635 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:57.924602    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5740efc6-1a2d-4762-93ad-ce5310f03c7a" path="/var/lib/kubelet/pods/5740efc6-1a2d-4762-93ad-ce5310f03c7a/volumes"
Oct 31 00:51:58.203954 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:58.203874    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1783fef-e971-4cce-861a-37b8c232d0b3-maas-api-tls\") pod \"maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl\" (UID: \"b1783fef-e971-4cce-861a-37b8c232d0b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl"
Oct 31 00:51:58.206298 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:58.206274    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1783fef-e971-4cce-861a-37b8c232d0b3-maas-api-tls\") pod \"maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl\" (UID: \"b1783fef-e971-4cce-861a-37b8c232d0b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl"
Oct 31 00:51:58.419907 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:58.419871    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl"
Oct 31 00:51:58.828401 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:58.828306    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl"]
Oct 31 00:51:58.853510 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:51:58.853463    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1783fef_e971_4cce_861a_37b8c232d0b3.slice/crio-5815b9af6225c4b1fdf2b847596d3b23ded9ca638f09ea3930ec7fbb4cc4c39b WatchSource:0}: Error finding container 5815b9af6225c4b1fdf2b847596d3b23ded9ca638f09ea3930ec7fbb4cc4c39b: Status 404 returned error can't find the container with id 5815b9af6225c4b1fdf2b847596d3b23ded9ca638f09ea3930ec7fbb4cc4c39b
Oct 31 00:51:59.319940 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.319902    4327 generic.go:356] "Generic (PLEG): container finished" podID="aca0da87-ad0a-45cc-a022-d4946e7fab22" containerID="af2e4eca3107c1082dd2626643c999ae9eb38e7f41b782c99e52982e2e64ad3f" exitCode=0
Oct 31 00:51:59.320123 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.319978    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh" event={"ID":"aca0da87-ad0a-45cc-a022-d4946e7fab22","Type":"ContainerDied","Data":"af2e4eca3107c1082dd2626643c999ae9eb38e7f41b782c99e52982e2e64ad3f"}
Oct 31 00:51:59.320123 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.320058    4327 scope.go:122] "RemoveContainer" containerID="2997c24959fba21be98fced109a4b85993b2dfc6151f35d7bc4c237aa3180251"
Oct 31 00:51:59.323276 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.323242    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl" event={"ID":"b1783fef-e971-4cce-861a-37b8c232d0b3","Type":"ContainerStarted","Data":"5815b9af6225c4b1fdf2b847596d3b23ded9ca638f09ea3930ec7fbb4cc4c39b"}
Oct 31 00:51:59.477776 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.477733    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:51:59.572058 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.572025    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-token\" (UniqueName: \"kubernetes.io/projected/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-token\") pod \"aca0da87-ad0a-45cc-a022-d4946e7fab22\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") "
Oct 31 00:51:59.572259 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.572183    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-credential-socket\") pod \"aca0da87-ad0a-45cc-a022-d4946e7fab22\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") "
Oct 31 00:51:59.572338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.572278    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-data\") pod \"aca0da87-ad0a-45cc-a022-d4946e7fab22\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") "
Oct 31 00:51:59.572393 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.572347    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aca0da87-ad0a-45cc-a022-d4946e7fab22-kube-api-access-hdcck\" (UniqueName: \"kubernetes.io/projected/aca0da87-ad0a-45cc-a022-d4946e7fab22-kube-api-access-hdcck\") pod \"aca0da87-ad0a-45cc-a022-d4946e7fab22\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") "
Oct 31 00:51:59.572448 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.572418    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-workload-socket\") pod \"aca0da87-ad0a-45cc-a022-d4946e7fab22\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") "
Oct 31 00:51:59.572501 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.572468    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-envoy\") pod \"aca0da87-ad0a-45cc-a022-d4946e7fab22\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") "
Oct 31 00:51:59.572554 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.572517    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aca0da87-ad0a-45cc-a022-d4946e7fab22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca0da87-ad0a-45cc-a022-d4946e7fab22-istiod-ca-cert\") pod \"aca0da87-ad0a-45cc-a022-d4946e7fab22\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") "
Oct 31 00:51:59.572611 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.572574    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-podinfo\") pod \"aca0da87-ad0a-45cc-a022-d4946e7fab22\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") "
Oct 31 00:51:59.572663 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.572611    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-data" pod "aca0da87-ad0a-45cc-a022-d4946e7fab22" (UID: "aca0da87-ad0a-45cc-a022-d4946e7fab22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:51:59.572785 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.572636    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-credential-socket" pod "aca0da87-ad0a-45cc-a022-d4946e7fab22" (UID: "aca0da87-ad0a-45cc-a022-d4946e7fab22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:51:59.572785 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.572708    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-workload-certs\") pod \"aca0da87-ad0a-45cc-a022-d4946e7fab22\" (UID: \"aca0da87-ad0a-45cc-a022-d4946e7fab22\") "
Oct 31 00:51:59.573072 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.572953    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-workload-socket" pod "aca0da87-ad0a-45cc-a022-d4946e7fab22" (UID: "aca0da87-ad0a-45cc-a022-d4946e7fab22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:51:59.573495 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.573304    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aca0da87-ad0a-45cc-a022-d4946e7fab22-istiod-ca-cert" pod "aca0da87-ad0a-45cc-a022-d4946e7fab22" (UID: "aca0da87-ad0a-45cc-a022-d4946e7fab22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:51:59.573570 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.573527    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-workload-certs" pod "aca0da87-ad0a-45cc-a022-d4946e7fab22" (UID: "aca0da87-ad0a-45cc-a022-d4946e7fab22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:51:59.573909 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.573821    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:51:59.573909 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.573846    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:51:59.573909 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.573859    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:51:59.573909 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.573871    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca0da87-ad0a-45cc-a022-d4946e7fab22-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:51:59.573909 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.573887    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:51:59.576195 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.576168    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-envoy" pod "aca0da87-ad0a-45cc-a022-d4946e7fab22" (UID: "aca0da87-ad0a-45cc-a022-d4946e7fab22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:51:59.576273 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.576238    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-token" pod "aca0da87-ad0a-45cc-a022-d4946e7fab22" (UID: "aca0da87-ad0a-45cc-a022-d4946e7fab22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:51:59.576330 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.576278    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca0da87-ad0a-45cc-a022-d4946e7fab22-kube-api-access-hdcck" pod "aca0da87-ad0a-45cc-a022-d4946e7fab22" (UID: "aca0da87-ad0a-45cc-a022-d4946e7fab22"). InnerVolumeSpecName "kube-api-access-hdcck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:51:59.576648 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.576630    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-podinfo" pod "aca0da87-ad0a-45cc-a022-d4946e7fab22" (UID: "aca0da87-ad0a-45cc-a022-d4946e7fab22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:51:59.675617 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.675581    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdcck\" (UniqueName: \"kubernetes.io/projected/aca0da87-ad0a-45cc-a022-d4946e7fab22-kube-api-access-hdcck\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:51:59.675617 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.675616    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:51:59.675858 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.675631    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:51:59.675858 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:51:59.675644    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aca0da87-ad0a-45cc-a022-d4946e7fab22-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:00.332490 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:00.332408    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"
Oct 31 00:52:00.332490 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:00.332434    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh" event={"ID":"aca0da87-ad0a-45cc-a022-d4946e7fab22","Type":"ContainerDied","Data":"bf7e8a8f8afe8f11ec978ea32a11ecdd00838eb269ce3096b816d21ae606ad8b"}
Oct 31 00:52:00.332490 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:00.332482    4327 scope.go:122] "RemoveContainer" containerID="af2e4eca3107c1082dd2626643c999ae9eb38e7f41b782c99e52982e2e64ad3f"
Oct 31 00:52:00.346294 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:00.346256    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl" event={"ID":"b1783fef-e971-4cce-861a-37b8c232d0b3","Type":"ContainerStarted","Data":"a2775b9ce46bd8bcb696c627a3cde8db6dc94a2c872814b1f16908dee43f43ed"}
Oct 31 00:52:00.346433 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:00.346359    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl"
Oct 31 00:52:00.372524 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:00.372492    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc"]
Oct 31 00:52:00.385135 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:00.385109    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5"]
Oct 31 00:52:00.424041 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:00.423986    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl" podStartSLOduration=3.013359066 podStartE2EDuration="3.423968099s" totalImagesPullingTime="410.609033ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:00.422459357 +0000 UTC m=+1980.644494913" watchObservedRunningTime="2026-10-31 00:52:00.423968099 +0000 UTC m=+1980.646003647"
Oct 31 00:52:00.479194 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:00.479145    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"]
Oct 31 00:52:00.484287 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:00.484246    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e166ca-openshift-default-6584656577-7cxfh"]
Oct 31 00:52:01.924426 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:01.924395    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aca0da87-ad0a-45cc-a022-d4946e7fab22" path="/var/lib/kubelet/pods/aca0da87-ad0a-45cc-a022-d4946e7fab22/volumes"
Oct 31 00:52:02.880517 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:02.880479    4327 scope.go:122] "RemoveContainer" containerID="525ba6907ee5358be6ec8b5b931b680e709ed66071c9f7eca1d88321b77343f2"
Oct 31 00:52:04.563840 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:04.563796    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"]
Oct 31 00:52:04.564260 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:04.564129    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6" podUID="14093068-2225-44ec-9ec6-9f85d5beeb4f" containerName="istio-proxy" containerID="cri-o://65fe431367a34165bbcc15c1b123d3ede06439585de31ecc2585fba2f00a5d4e" gracePeriod=30
Oct 31 00:52:05.048063 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:05.048031    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65"]
Oct 31 00:52:05.108771 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:05.108735    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5"]
Oct 31 00:52:05.373910 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:05.373829    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc" podUID="67126c73-73e2-40d7-acd8-593ac959261d" containerName="payload-processing" containerID="cri-o://50b5f98b221df7b149d50c9d81a7aeac024f7a2b9a24f9213b7df469bc0e204a" gracePeriod=25
Oct 31 00:52:05.385394 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:05.385351    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5" podUID="a114e7ea-38c3-41c9-8a54-7bcbb47ca591" containerName="payload-pre-processing" containerID="cri-o://991ea598e0397d613c14a3a976209230c5c3192122d20f287af42577f6c53087" gracePeriod=25
Oct 31 00:52:05.829171 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:05.829146    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc"
Oct 31 00:52:05.860341 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:05.860314    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5"
Oct 31 00:52:05.960900 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:05.946342    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a114e7ea-38c3-41c9-8a54-7bcbb47ca591-kube-api-access-rmn67\" (UniqueName: \"kubernetes.io/projected/a114e7ea-38c3-41c9-8a54-7bcbb47ca591-kube-api-access-rmn67\") pod \"a114e7ea-38c3-41c9-8a54-7bcbb47ca591\" (UID: \"a114e7ea-38c3-41c9-8a54-7bcbb47ca591\") "
Oct 31 00:52:05.960900 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:05.946431    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67126c73-73e2-40d7-acd8-593ac959261d-kube-api-access-tlfds\" (UniqueName: \"kubernetes.io/projected/67126c73-73e2-40d7-acd8-593ac959261d-kube-api-access-tlfds\") pod \"67126c73-73e2-40d7-acd8-593ac959261d\" (UID: \"67126c73-73e2-40d7-acd8-593ac959261d\") "
Oct 31 00:52:05.960900 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:05.946573    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67126c73-73e2-40d7-acd8-593ac959261d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67126c73-73e2-40d7-acd8-593ac959261d-plugins-config-volume\") pod \"67126c73-73e2-40d7-acd8-593ac959261d\" (UID: \"67126c73-73e2-40d7-acd8-593ac959261d\") "
Oct 31 00:52:05.960900 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:05.946829    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a114e7ea-38c3-41c9-8a54-7bcbb47ca591-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a114e7ea-38c3-41c9-8a54-7bcbb47ca591-plugins-config-volume\") pod \"a114e7ea-38c3-41c9-8a54-7bcbb47ca591\" (UID: \"a114e7ea-38c3-41c9-8a54-7bcbb47ca591\") "
Oct 31 00:52:05.960900 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:05.948372    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a114e7ea-38c3-41c9-8a54-7bcbb47ca591-plugins-config-volume" pod "a114e7ea-38c3-41c9-8a54-7bcbb47ca591" (UID: "a114e7ea-38c3-41c9-8a54-7bcbb47ca591"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:52:05.960900 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:05.959705    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67126c73-73e2-40d7-acd8-593ac959261d-plugins-config-volume" pod "67126c73-73e2-40d7-acd8-593ac959261d" (UID: "67126c73-73e2-40d7-acd8-593ac959261d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:52:05.977354 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:05.976259    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67126c73-73e2-40d7-acd8-593ac959261d-kube-api-access-tlfds" pod "67126c73-73e2-40d7-acd8-593ac959261d" (UID: "67126c73-73e2-40d7-acd8-593ac959261d"). InnerVolumeSpecName "kube-api-access-tlfds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:52:06.002753 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:05.985856    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a114e7ea-38c3-41c9-8a54-7bcbb47ca591-kube-api-access-rmn67" pod "a114e7ea-38c3-41c9-8a54-7bcbb47ca591" (UID: "a114e7ea-38c3-41c9-8a54-7bcbb47ca591"). InnerVolumeSpecName "kube-api-access-rmn67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:52:06.130761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.124491    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a114e7ea-38c3-41c9-8a54-7bcbb47ca591-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:06.130761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.124535    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmn67\" (UniqueName: \"kubernetes.io/projected/a114e7ea-38c3-41c9-8a54-7bcbb47ca591-kube-api-access-rmn67\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:06.130761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.124551    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlfds\" (UniqueName: \"kubernetes.io/projected/67126c73-73e2-40d7-acd8-593ac959261d-kube-api-access-tlfds\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:06.130761 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.124565    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67126c73-73e2-40d7-acd8-593ac959261d-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:06.386563 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.386535    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl"
Oct 31 00:52:06.435423 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.435382    4327 generic.go:356] "Generic (PLEG): container finished" podID="a114e7ea-38c3-41c9-8a54-7bcbb47ca591" containerID="991ea598e0397d613c14a3a976209230c5c3192122d20f287af42577f6c53087" exitCode=0
Oct 31 00:52:06.435600 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.435419    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5" event={"ID":"a114e7ea-38c3-41c9-8a54-7bcbb47ca591","Type":"ContainerDied","Data":"991ea598e0397d613c14a3a976209230c5c3192122d20f287af42577f6c53087"}
Oct 31 00:52:06.435600 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.435480    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5" event={"ID":"a114e7ea-38c3-41c9-8a54-7bcbb47ca591","Type":"ContainerDied","Data":"3f3fad47baf27473998d0b280343c8c74f4c769f07930a96b39427aebec8e95f"}
Oct 31 00:52:06.435600 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.435502    4327 scope.go:122] "RemoveContainer" containerID="991ea598e0397d613c14a3a976209230c5c3192122d20f287af42577f6c53087"
Oct 31 00:52:06.435600 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.435432    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5"
Oct 31 00:52:06.437499 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.437474    4327 generic.go:356] "Generic (PLEG): container finished" podID="67126c73-73e2-40d7-acd8-593ac959261d" containerID="50b5f98b221df7b149d50c9d81a7aeac024f7a2b9a24f9213b7df469bc0e204a" exitCode=0
Oct 31 00:52:06.437618 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.437514    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc"
Oct 31 00:52:06.437618 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.437546    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc" event={"ID":"67126c73-73e2-40d7-acd8-593ac959261d","Type":"ContainerDied","Data":"50b5f98b221df7b149d50c9d81a7aeac024f7a2b9a24f9213b7df469bc0e204a"}
Oct 31 00:52:06.437618 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.437586    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc" event={"ID":"67126c73-73e2-40d7-acd8-593ac959261d","Type":"ContainerDied","Data":"246750c9eee220746c55dafa9e57d51317cc1548c4fdde7ff9b00f9909a84cf5"}
Oct 31 00:52:06.449434 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.449410    4327 scope.go:122] "RemoveContainer" containerID="991ea598e0397d613c14a3a976209230c5c3192122d20f287af42577f6c53087"
Oct 31 00:52:06.450140 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:52:06.450111    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"991ea598e0397d613c14a3a976209230c5c3192122d20f287af42577f6c53087\": container with ID starting with 991ea598e0397d613c14a3a976209230c5c3192122d20f287af42577f6c53087 not found: ID does not exist" containerID="991ea598e0397d613c14a3a976209230c5c3192122d20f287af42577f6c53087"
Oct 31 00:52:06.450239 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.450149    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"991ea598e0397d613c14a3a976209230c5c3192122d20f287af42577f6c53087"} err="failed to get container status \"991ea598e0397d613c14a3a976209230c5c3192122d20f287af42577f6c53087\": rpc error: code = NotFound desc = could not find container \"991ea598e0397d613c14a3a976209230c5c3192122d20f287af42577f6c53087\": container with ID starting with 991ea598e0397d613c14a3a976209230c5c3192122d20f287af42577f6c53087 not found: ID does not exist"
Oct 31 00:52:06.450239 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.450174    4327 scope.go:122] "RemoveContainer" containerID="50b5f98b221df7b149d50c9d81a7aeac024f7a2b9a24f9213b7df469bc0e204a"
Oct 31 00:52:06.464028 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.464000    4327 scope.go:122] "RemoveContainer" containerID="50b5f98b221df7b149d50c9d81a7aeac024f7a2b9a24f9213b7df469bc0e204a"
Oct 31 00:52:06.464586 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:52:06.464552    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50b5f98b221df7b149d50c9d81a7aeac024f7a2b9a24f9213b7df469bc0e204a\": container with ID starting with 50b5f98b221df7b149d50c9d81a7aeac024f7a2b9a24f9213b7df469bc0e204a not found: ID does not exist" containerID="50b5f98b221df7b149d50c9d81a7aeac024f7a2b9a24f9213b7df469bc0e204a"
Oct 31 00:52:06.464753 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.464597    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50b5f98b221df7b149d50c9d81a7aeac024f7a2b9a24f9213b7df469bc0e204a"} err="failed to get container status \"50b5f98b221df7b149d50c9d81a7aeac024f7a2b9a24f9213b7df469bc0e204a\": rpc error: code = NotFound desc = could not find container \"50b5f98b221df7b149d50c9d81a7aeac024f7a2b9a24f9213b7df469bc0e204a\": container with ID starting with 50b5f98b221df7b149d50c9d81a7aeac024f7a2b9a24f9213b7df469bc0e204a not found: ID does not exist"
Oct 31 00:52:06.499798 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.499747    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5"]
Oct 31 00:52:06.507049 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.507004    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-tlrt5"]
Oct 31 00:52:06.536520 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.536489    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc"]
Oct 31 00:52:06.541125 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.541097    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-jmtjc"]
Oct 31 00:52:06.938746 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.938714    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv"]
Oct 31 00:52:06.939623 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.939584    4327 cpu_manager.go:414] "containerMap: removing container" podUID="67126c73-73e2-40d7-acd8-593ac959261d" containerName="payload-processing"
Oct 31 00:52:06.939623 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.939608    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67126c73-73e2-40d7-acd8-593ac959261d" containerName="payload-processing"
Oct 31 00:52:06.939623 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.939625    4327 cpu_manager.go:414] "containerMap: removing container" podUID="a114e7ea-38c3-41c9-8a54-7bcbb47ca591" containerName="payload-pre-processing"
Oct 31 00:52:06.939623 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.939632    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a114e7ea-38c3-41c9-8a54-7bcbb47ca591" containerName="payload-pre-processing"
Oct 31 00:52:06.939949 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.939649    4327 cpu_manager.go:414] "containerMap: removing container" podUID="aca0da87-ad0a-45cc-a022-d4946e7fab22" containerName="istio-proxy"
Oct 31 00:52:06.939949 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.939656    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aca0da87-ad0a-45cc-a022-d4946e7fab22" containerName="istio-proxy"
Oct 31 00:52:06.939949 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.939721    4327 cpu_manager.go:414] "containerMap: removing container" podUID="aca0da87-ad0a-45cc-a022-d4946e7fab22" containerName="istio-proxy"
Oct 31 00:52:06.939949 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.939727    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aca0da87-ad0a-45cc-a022-d4946e7fab22" containerName="istio-proxy"
Oct 31 00:52:06.939949 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.939880    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="aca0da87-ad0a-45cc-a022-d4946e7fab22" containerName="istio-proxy"
Oct 31 00:52:06.939949 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.939892    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="67126c73-73e2-40d7-acd8-593ac959261d" containerName="payload-processing"
Oct 31 00:52:06.939949 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.939903    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="a114e7ea-38c3-41c9-8a54-7bcbb47ca591" containerName="payload-pre-processing"
Oct 31 00:52:06.945276 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.945259    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv"
Oct 31 00:52:06.973623 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.973593    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv"]
Oct 31 00:52:06.974878 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:06.974853    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-fd1dfc-serving-cert\""
Oct 31 00:52:07.025576 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:07.025532    4327 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:52:07.025806 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:07.025611    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6" podUID="14093068-2225-44ec-9ec6-9f85d5beeb4f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:52:07.038393 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:07.038355    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85wp9\" (UniqueName: \"kubernetes.io/projected/75942e65-46b1-4ec3-84a7-425e74bd1296-kube-api-access-85wp9\") pod \"maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv\" (UID: \"75942e65-46b1-4ec3-84a7-425e74bd1296\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv"
Oct 31 00:52:07.038606 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:07.038584    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75942e65-46b1-4ec3-84a7-425e74bd1296-maas-api-tls\") pod \"maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv\" (UID: \"75942e65-46b1-4ec3-84a7-425e74bd1296\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv"
Oct 31 00:52:07.039599 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:07.038839    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75942e65-46b1-4ec3-84a7-425e74bd1296-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv\" (UID: \"75942e65-46b1-4ec3-84a7-425e74bd1296\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv"
Oct 31 00:52:07.039599 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:07.038887    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75942e65-46b1-4ec3-84a7-425e74bd1296-metrics-certs\") pod \"maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv\" (UID: \"75942e65-46b1-4ec3-84a7-425e74bd1296\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv"
Oct 31 00:52:07.140416 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:07.140380    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75942e65-46b1-4ec3-84a7-425e74bd1296-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv\" (UID: \"75942e65-46b1-4ec3-84a7-425e74bd1296\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv"
Oct 31 00:52:07.140416 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:07.140425    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75942e65-46b1-4ec3-84a7-425e74bd1296-metrics-certs\") pod \"maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv\" (UID: \"75942e65-46b1-4ec3-84a7-425e74bd1296\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv"
Oct 31 00:52:07.140627 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:07.140556    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85wp9\" (UniqueName: \"kubernetes.io/projected/75942e65-46b1-4ec3-84a7-425e74bd1296-kube-api-access-85wp9\") pod \"maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv\" (UID: \"75942e65-46b1-4ec3-84a7-425e74bd1296\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv"
Oct 31 00:52:07.140733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:07.140708    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75942e65-46b1-4ec3-84a7-425e74bd1296-maas-api-tls\") pod \"maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv\" (UID: \"75942e65-46b1-4ec3-84a7-425e74bd1296\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv"
Oct 31 00:52:07.141128 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:07.141105    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75942e65-46b1-4ec3-84a7-425e74bd1296-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv\" (UID: \"75942e65-46b1-4ec3-84a7-425e74bd1296\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv"
Oct 31 00:52:07.142916 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:07.142898    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75942e65-46b1-4ec3-84a7-425e74bd1296-metrics-certs\") pod \"maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv\" (UID: \"75942e65-46b1-4ec3-84a7-425e74bd1296\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv"
Oct 31 00:52:07.143070 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:07.143051    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75942e65-46b1-4ec3-84a7-425e74bd1296-maas-api-tls\") pod \"maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv\" (UID: \"75942e65-46b1-4ec3-84a7-425e74bd1296\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv"
Oct 31 00:52:07.207973 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:07.206761    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85wp9\" (UniqueName: \"kubernetes.io/projected/75942e65-46b1-4ec3-84a7-425e74bd1296-kube-api-access-85wp9\") pod \"maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv\" (UID: \"75942e65-46b1-4ec3-84a7-425e74bd1296\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv"
Oct 31 00:52:07.269070 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:07.266902    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv"
Oct 31 00:52:07.634942 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:07.634900    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv"]
Oct 31 00:52:07.636653 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:52:07.636625    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75942e65_46b1_4ec3_84a7_425e74bd1296.slice/crio-68e901a237fb79541d3d9b7bca83536ad372e3fe246cbc617b62e8226bd02083 WatchSource:0}: Error finding container 68e901a237fb79541d3d9b7bca83536ad372e3fe246cbc617b62e8226bd02083: Status 404 returned error can't find the container with id 68e901a237fb79541d3d9b7bca83536ad372e3fe246cbc617b62e8226bd02083
Oct 31 00:52:07.924722 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:07.924605    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67126c73-73e2-40d7-acd8-593ac959261d" path="/var/lib/kubelet/pods/67126c73-73e2-40d7-acd8-593ac959261d/volumes"
Oct 31 00:52:07.925186 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:07.925169    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a114e7ea-38c3-41c9-8a54-7bcbb47ca591" path="/var/lib/kubelet/pods/a114e7ea-38c3-41c9-8a54-7bcbb47ca591/volumes"
Oct 31 00:52:08.459746 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:08.459710    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv" event={"ID":"75942e65-46b1-4ec3-84a7-425e74bd1296","Type":"ContainerStarted","Data":"24b73a577c0306d7e800a374e43f612308b69b4c9df9e294aeb415d12148306b"}
Oct 31 00:52:08.460801 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:08.460757    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv" event={"ID":"75942e65-46b1-4ec3-84a7-425e74bd1296","Type":"ContainerStarted","Data":"68e901a237fb79541d3d9b7bca83536ad372e3fe246cbc617b62e8226bd02083"}
Oct 31 00:52:08.462132 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:08.462080    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv"
Oct 31 00:52:08.516425 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:08.516369    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv" podStartSLOduration=2.134476839 podStartE2EDuration="2.516354943s" totalImagesPullingTime="381.878104ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:08.510375019 +0000 UTC m=+1988.732410563" watchObservedRunningTime="2026-10-31 00:52:08.516354943 +0000 UTC m=+1988.738390485"
Oct 31 00:52:09.860008 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.859980    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:52:09.894585 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.894552    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14093068-2225-44ec-9ec6-9f85d5beeb4f-kube-api-access-9gztr\" (UniqueName: \"kubernetes.io/projected/14093068-2225-44ec-9ec6-9f85d5beeb4f-kube-api-access-9gztr\") pod \"14093068-2225-44ec-9ec6-9f85d5beeb4f\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") "
Oct 31 00:52:09.894776 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.894595    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14093068-2225-44ec-9ec6-9f85d5beeb4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14093068-2225-44ec-9ec6-9f85d5beeb4f-istiod-ca-cert\") pod \"14093068-2225-44ec-9ec6-9f85d5beeb4f\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") "
Oct 31 00:52:09.894776 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.894628    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-data\") pod \"14093068-2225-44ec-9ec6-9f85d5beeb4f\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") "
Oct 31 00:52:09.894916 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.894796    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-podinfo\") pod \"14093068-2225-44ec-9ec6-9f85d5beeb4f\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") "
Oct 31 00:52:09.894916 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.894845    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-credential-socket\") pod \"14093068-2225-44ec-9ec6-9f85d5beeb4f\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") "
Oct 31 00:52:09.894916 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.894894    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-envoy\") pod \"14093068-2225-44ec-9ec6-9f85d5beeb4f\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") "
Oct 31 00:52:09.895072 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.894950    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-workload-certs\") pod \"14093068-2225-44ec-9ec6-9f85d5beeb4f\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") "
Oct 31 00:52:09.895072 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.894953    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-data" pod "14093068-2225-44ec-9ec6-9f85d5beeb4f" (UID: "14093068-2225-44ec-9ec6-9f85d5beeb4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:52:09.895072 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.895020    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-workload-socket\") pod \"14093068-2225-44ec-9ec6-9f85d5beeb4f\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") "
Oct 31 00:52:09.895072 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.895039    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14093068-2225-44ec-9ec6-9f85d5beeb4f-istiod-ca-cert" pod "14093068-2225-44ec-9ec6-9f85d5beeb4f" (UID: "14093068-2225-44ec-9ec6-9f85d5beeb4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:52:09.895288 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.895081    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-token\" (UniqueName: \"kubernetes.io/projected/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-token\") pod \"14093068-2225-44ec-9ec6-9f85d5beeb4f\" (UID: \"14093068-2225-44ec-9ec6-9f85d5beeb4f\") "
Oct 31 00:52:09.895288 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.895178    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-credential-socket" pod "14093068-2225-44ec-9ec6-9f85d5beeb4f" (UID: "14093068-2225-44ec-9ec6-9f85d5beeb4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:52:09.895288 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.895274    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-workload-certs" pod "14093068-2225-44ec-9ec6-9f85d5beeb4f" (UID: "14093068-2225-44ec-9ec6-9f85d5beeb4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:52:09.895438 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.895343    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-workload-socket" pod "14093068-2225-44ec-9ec6-9f85d5beeb4f" (UID: "14093068-2225-44ec-9ec6-9f85d5beeb4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:52:09.895746 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.895725    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14093068-2225-44ec-9ec6-9f85d5beeb4f-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:09.895746 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.895744    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:09.895913 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.895754    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:09.895913 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.895763    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:09.895913 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.895771    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:09.897095 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.897066    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-podinfo" pod "14093068-2225-44ec-9ec6-9f85d5beeb4f" (UID: "14093068-2225-44ec-9ec6-9f85d5beeb4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:52:09.897461 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.897437    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14093068-2225-44ec-9ec6-9f85d5beeb4f-kube-api-access-9gztr" pod "14093068-2225-44ec-9ec6-9f85d5beeb4f" (UID: "14093068-2225-44ec-9ec6-9f85d5beeb4f"). InnerVolumeSpecName "kube-api-access-9gztr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:52:09.897653 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.897627    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-envoy" pod "14093068-2225-44ec-9ec6-9f85d5beeb4f" (UID: "14093068-2225-44ec-9ec6-9f85d5beeb4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:52:09.897790 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.897775    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-token" pod "14093068-2225-44ec-9ec6-9f85d5beeb4f" (UID: "14093068-2225-44ec-9ec6-9f85d5beeb4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:52:09.997882 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.997839    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:09.997882 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.997876    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:09.998086 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.997890    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14093068-2225-44ec-9ec6-9f85d5beeb4f-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:09.998086 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:09.997904    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gztr\" (UniqueName: \"kubernetes.io/projected/14093068-2225-44ec-9ec6-9f85d5beeb4f-kube-api-access-9gztr\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:10.048362 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.048317    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65" podUID="39a92afd-325e-4e33-9d0c-6b9fde7a49c3" containerName="payload-processing" containerID="cri-o://8eba09a60b769dd64a38906d0984877a8df27a1ff98201103a3b55b2309b8a9b" gracePeriod=25
Oct 31 00:52:10.109079 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.109038    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5" podUID="9751c672-027d-40c3-9da4-a18c4671f8fa" containerName="payload-pre-processing" containerID="cri-o://386f2360276b42100359c93542d91148ae8db81181178a25a90d6302b8d09849" gracePeriod=25
Oct 31 00:52:10.375775 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.375748    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65"
Oct 31 00:52:10.403347 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.403311    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39a92afd-325e-4e33-9d0c-6b9fde7a49c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39a92afd-325e-4e33-9d0c-6b9fde7a49c3-plugins-config-volume\") pod \"39a92afd-325e-4e33-9d0c-6b9fde7a49c3\" (UID: \"39a92afd-325e-4e33-9d0c-6b9fde7a49c3\") "
Oct 31 00:52:10.403546 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.403371    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39a92afd-325e-4e33-9d0c-6b9fde7a49c3-kube-api-access-gsqdx\" (UniqueName: \"kubernetes.io/projected/39a92afd-325e-4e33-9d0c-6b9fde7a49c3-kube-api-access-gsqdx\") pod \"39a92afd-325e-4e33-9d0c-6b9fde7a49c3\" (UID: \"39a92afd-325e-4e33-9d0c-6b9fde7a49c3\") "
Oct 31 00:52:10.404075 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.404045    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39a92afd-325e-4e33-9d0c-6b9fde7a49c3-plugins-config-volume" pod "39a92afd-325e-4e33-9d0c-6b9fde7a49c3" (UID: "39a92afd-325e-4e33-9d0c-6b9fde7a49c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:52:10.405774 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.405744    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a92afd-325e-4e33-9d0c-6b9fde7a49c3-kube-api-access-gsqdx" pod "39a92afd-325e-4e33-9d0c-6b9fde7a49c3" (UID: "39a92afd-325e-4e33-9d0c-6b9fde7a49c3"). InnerVolumeSpecName "kube-api-access-gsqdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:52:10.436585 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.436558    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5"
Oct 31 00:52:10.484023 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.483522    4327 generic.go:356] "Generic (PLEG): container finished" podID="14093068-2225-44ec-9ec6-9f85d5beeb4f" containerID="65fe431367a34165bbcc15c1b123d3ede06439585de31ecc2585fba2f00a5d4e" exitCode=0
Oct 31 00:52:10.484023 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.483602    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"
Oct 31 00:52:10.484023 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.483855    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6" event={"ID":"14093068-2225-44ec-9ec6-9f85d5beeb4f","Type":"ContainerDied","Data":"65fe431367a34165bbcc15c1b123d3ede06439585de31ecc2585fba2f00a5d4e"}
Oct 31 00:52:10.484023 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.483911    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6" event={"ID":"14093068-2225-44ec-9ec6-9f85d5beeb4f","Type":"ContainerDied","Data":"4e76d5e7824c01fb8535c790a26915f155f5a0ed253d4ceef7d4b985ea1405eb"}
Oct 31 00:52:10.484023 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.483943    4327 scope.go:122] "RemoveContainer" containerID="65fe431367a34165bbcc15c1b123d3ede06439585de31ecc2585fba2f00a5d4e"
Oct 31 00:52:10.490219 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.490188    4327 generic.go:356] "Generic (PLEG): container finished" podID="39a92afd-325e-4e33-9d0c-6b9fde7a49c3" containerID="8eba09a60b769dd64a38906d0984877a8df27a1ff98201103a3b55b2309b8a9b" exitCode=0
Oct 31 00:52:10.490409 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.490387    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65" event={"ID":"39a92afd-325e-4e33-9d0c-6b9fde7a49c3","Type":"ContainerDied","Data":"8eba09a60b769dd64a38906d0984877a8df27a1ff98201103a3b55b2309b8a9b"}
Oct 31 00:52:10.490493 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.490441    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65" event={"ID":"39a92afd-325e-4e33-9d0c-6b9fde7a49c3","Type":"ContainerDied","Data":"df4a71c3e7079d3f01f02468a25e636954a9900f25911680acf03ca36b1e8cb8"}
Oct 31 00:52:10.490550 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.490534    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65"
Oct 31 00:52:10.495271 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.495241    4327 generic.go:356] "Generic (PLEG): container finished" podID="9751c672-027d-40c3-9da4-a18c4671f8fa" containerID="386f2360276b42100359c93542d91148ae8db81181178a25a90d6302b8d09849" exitCode=0
Oct 31 00:52:10.495800 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.495779    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5"
Oct 31 00:52:10.495926 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.495907    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5" event={"ID":"9751c672-027d-40c3-9da4-a18c4671f8fa","Type":"ContainerDied","Data":"386f2360276b42100359c93542d91148ae8db81181178a25a90d6302b8d09849"}
Oct 31 00:52:10.496002 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.495951    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5" event={"ID":"9751c672-027d-40c3-9da4-a18c4671f8fa","Type":"ContainerDied","Data":"3aae15d180e71aa3c0387f23aa9cfdfc4cf7c2981a20d37943c07348110cf02d"}
Oct 31 00:52:10.504725 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.504697    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9751c672-027d-40c3-9da4-a18c4671f8fa-kube-api-access-4cmjr\" (UniqueName: \"kubernetes.io/projected/9751c672-027d-40c3-9da4-a18c4671f8fa-kube-api-access-4cmjr\") pod \"9751c672-027d-40c3-9da4-a18c4671f8fa\" (UID: \"9751c672-027d-40c3-9da4-a18c4671f8fa\") "
Oct 31 00:52:10.504893 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.504874    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9751c672-027d-40c3-9da4-a18c4671f8fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9751c672-027d-40c3-9da4-a18c4671f8fa-plugins-config-volume\") pod \"9751c672-027d-40c3-9da4-a18c4671f8fa\" (UID: \"9751c672-027d-40c3-9da4-a18c4671f8fa\") "
Oct 31 00:52:10.505530 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.505496    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9751c672-027d-40c3-9da4-a18c4671f8fa-plugins-config-volume" pod "9751c672-027d-40c3-9da4-a18c4671f8fa" (UID: "9751c672-027d-40c3-9da4-a18c4671f8fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:52:10.505772 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.505724    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39a92afd-325e-4e33-9d0c-6b9fde7a49c3-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:10.505772 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.505748    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsqdx\" (UniqueName: \"kubernetes.io/projected/39a92afd-325e-4e33-9d0c-6b9fde7a49c3-kube-api-access-gsqdx\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:10.505772 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.505764    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9751c672-027d-40c3-9da4-a18c4671f8fa-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:10.510868 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.510844    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9751c672-027d-40c3-9da4-a18c4671f8fa-kube-api-access-4cmjr" pod "9751c672-027d-40c3-9da4-a18c4671f8fa" (UID: "9751c672-027d-40c3-9da4-a18c4671f8fa"). InnerVolumeSpecName "kube-api-access-4cmjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:52:10.510980 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.510868    4327 scope.go:122] "RemoveContainer" containerID="e0d4e6a9ebc0909ff8a1f19cdac60d9c215a7804f55bfc40cfdef0a915dbcf8a"
Oct 31 00:52:10.525764 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.525738    4327 scope.go:122] "RemoveContainer" containerID="65fe431367a34165bbcc15c1b123d3ede06439585de31ecc2585fba2f00a5d4e"
Oct 31 00:52:10.538102 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:52:10.538062    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65fe431367a34165bbcc15c1b123d3ede06439585de31ecc2585fba2f00a5d4e\": container with ID starting with 65fe431367a34165bbcc15c1b123d3ede06439585de31ecc2585fba2f00a5d4e not found: ID does not exist" containerID="65fe431367a34165bbcc15c1b123d3ede06439585de31ecc2585fba2f00a5d4e"
Oct 31 00:52:10.538274 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.538110    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65fe431367a34165bbcc15c1b123d3ede06439585de31ecc2585fba2f00a5d4e"} err="failed to get container status \"65fe431367a34165bbcc15c1b123d3ede06439585de31ecc2585fba2f00a5d4e\": rpc error: code = NotFound desc = could not find container \"65fe431367a34165bbcc15c1b123d3ede06439585de31ecc2585fba2f00a5d4e\": container with ID starting with 65fe431367a34165bbcc15c1b123d3ede06439585de31ecc2585fba2f00a5d4e not found: ID does not exist"
Oct 31 00:52:10.538274 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.538137    4327 scope.go:122] "RemoveContainer" containerID="e0d4e6a9ebc0909ff8a1f19cdac60d9c215a7804f55bfc40cfdef0a915dbcf8a"
Oct 31 00:52:10.538730 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:52:10.538701    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0d4e6a9ebc0909ff8a1f19cdac60d9c215a7804f55bfc40cfdef0a915dbcf8a\": container with ID starting with e0d4e6a9ebc0909ff8a1f19cdac60d9c215a7804f55bfc40cfdef0a915dbcf8a not found: ID does not exist" containerID="e0d4e6a9ebc0909ff8a1f19cdac60d9c215a7804f55bfc40cfdef0a915dbcf8a"
Oct 31 00:52:10.538837 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.538738    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0d4e6a9ebc0909ff8a1f19cdac60d9c215a7804f55bfc40cfdef0a915dbcf8a"} err="failed to get container status \"e0d4e6a9ebc0909ff8a1f19cdac60d9c215a7804f55bfc40cfdef0a915dbcf8a\": rpc error: code = NotFound desc = could not find container \"e0d4e6a9ebc0909ff8a1f19cdac60d9c215a7804f55bfc40cfdef0a915dbcf8a\": container with ID starting with e0d4e6a9ebc0909ff8a1f19cdac60d9c215a7804f55bfc40cfdef0a915dbcf8a not found: ID does not exist"
Oct 31 00:52:10.538837 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.538767    4327 scope.go:122] "RemoveContainer" containerID="8eba09a60b769dd64a38906d0984877a8df27a1ff98201103a3b55b2309b8a9b"
Oct 31 00:52:10.584622 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.584278    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"]
Oct 31 00:52:10.593644 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.588818    4327 scope.go:122] "RemoveContainer" containerID="8eba09a60b769dd64a38906d0984877a8df27a1ff98201103a3b55b2309b8a9b"
Oct 31 00:52:10.594459 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:52:10.594421    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eba09a60b769dd64a38906d0984877a8df27a1ff98201103a3b55b2309b8a9b\": container with ID starting with 8eba09a60b769dd64a38906d0984877a8df27a1ff98201103a3b55b2309b8a9b not found: ID does not exist" containerID="8eba09a60b769dd64a38906d0984877a8df27a1ff98201103a3b55b2309b8a9b"
Oct 31 00:52:10.594640 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.594615    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eba09a60b769dd64a38906d0984877a8df27a1ff98201103a3b55b2309b8a9b"} err="failed to get container status \"8eba09a60b769dd64a38906d0984877a8df27a1ff98201103a3b55b2309b8a9b\": rpc error: code = NotFound desc = could not find container \"8eba09a60b769dd64a38906d0984877a8df27a1ff98201103a3b55b2309b8a9b\": container with ID starting with 8eba09a60b769dd64a38906d0984877a8df27a1ff98201103a3b55b2309b8a9b not found: ID does not exist"
Oct 31 00:52:10.594790 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.594773    4327 scope.go:122] "RemoveContainer" containerID="386f2360276b42100359c93542d91148ae8db81181178a25a90d6302b8d09849"
Oct 31 00:52:10.596195 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.596168    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nn2c6"]
Oct 31 00:52:10.630219 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.624577    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cmjr\" (UniqueName: \"kubernetes.io/projected/9751c672-027d-40c3-9da4-a18c4671f8fa-kube-api-access-4cmjr\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:10.667885 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.667849    4327 scope.go:122] "RemoveContainer" containerID="386f2360276b42100359c93542d91148ae8db81181178a25a90d6302b8d09849"
Oct 31 00:52:10.675846 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:52:10.675737    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"386f2360276b42100359c93542d91148ae8db81181178a25a90d6302b8d09849\": container with ID starting with 386f2360276b42100359c93542d91148ae8db81181178a25a90d6302b8d09849 not found: ID does not exist" containerID="386f2360276b42100359c93542d91148ae8db81181178a25a90d6302b8d09849"
Oct 31 00:52:10.675846 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.675789    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386f2360276b42100359c93542d91148ae8db81181178a25a90d6302b8d09849"} err="failed to get container status \"386f2360276b42100359c93542d91148ae8db81181178a25a90d6302b8d09849\": rpc error: code = NotFound desc = could not find container \"386f2360276b42100359c93542d91148ae8db81181178a25a90d6302b8d09849\": container with ID starting with 386f2360276b42100359c93542d91148ae8db81181178a25a90d6302b8d09849 not found: ID does not exist"
Oct 31 00:52:10.678023 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.677991    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65"]
Oct 31 00:52:10.681726 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.681696    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3d21c7-588788f9c-h9x65"]
Oct 31 00:52:10.846404 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.846376    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5"]
Oct 31 00:52:10.866587 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:10.866548    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d21c7-65ffb6f459-d9gb5"]
Oct 31 00:52:11.923350 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:11.923319    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14093068-2225-44ec-9ec6-9f85d5beeb4f" path="/var/lib/kubelet/pods/14093068-2225-44ec-9ec6-9f85d5beeb4f/volumes"
Oct 31 00:52:11.923856 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:11.923841    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39a92afd-325e-4e33-9d0c-6b9fde7a49c3" path="/var/lib/kubelet/pods/39a92afd-325e-4e33-9d0c-6b9fde7a49c3/volumes"
Oct 31 00:52:11.924243 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:11.924230    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9751c672-027d-40c3-9da4-a18c4671f8fa" path="/var/lib/kubelet/pods/9751c672-027d-40c3-9da4-a18c4671f8fa/volumes"
Oct 31 00:52:13.582889 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:13.582855    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s"]
Oct 31 00:52:13.583311 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:13.583089    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s" podUID="ae6fa740-5a10-4bd4-88cd-51f4006e79f5" containerName="maas-api" containerID="cri-o://374cd93c64cfa4778de289d6fe1b02af6f28b87f13cd6552f2bb443ecb82209a" gracePeriod=30
Oct 31 00:52:14.263702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:14.263074    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s"
Oct 31 00:52:14.413231 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:14.413197    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-openshift-service-ca\") pod \"ae6fa740-5a10-4bd4-88cd-51f4006e79f5\" (UID: \"ae6fa740-5a10-4bd4-88cd-51f4006e79f5\") "
Oct 31 00:52:14.413412 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:14.413289    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-metrics-certs\") pod \"ae6fa740-5a10-4bd4-88cd-51f4006e79f5\" (UID: \"ae6fa740-5a10-4bd4-88cd-51f4006e79f5\") "
Oct 31 00:52:14.413412 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:14.413394    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-maas-api-tls\") pod \"ae6fa740-5a10-4bd4-88cd-51f4006e79f5\" (UID: \"ae6fa740-5a10-4bd4-88cd-51f4006e79f5\") "
Oct 31 00:52:14.413537 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:14.413422    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-kube-api-access-drdwj\" (UniqueName: \"kubernetes.io/projected/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-kube-api-access-drdwj\") pod \"ae6fa740-5a10-4bd4-88cd-51f4006e79f5\" (UID: \"ae6fa740-5a10-4bd4-88cd-51f4006e79f5\") "
Oct 31 00:52:14.413983 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:14.413949    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-openshift-service-ca" pod "ae6fa740-5a10-4bd4-88cd-51f4006e79f5" (UID: "ae6fa740-5a10-4bd4-88cd-51f4006e79f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:52:14.414358 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:14.414330    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:14.415884 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:14.415847    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-metrics-certs" pod "ae6fa740-5a10-4bd4-88cd-51f4006e79f5" (UID: "ae6fa740-5a10-4bd4-88cd-51f4006e79f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:52:14.416388 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:14.416360    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-kube-api-access-drdwj" pod "ae6fa740-5a10-4bd4-88cd-51f4006e79f5" (UID: "ae6fa740-5a10-4bd4-88cd-51f4006e79f5"). InnerVolumeSpecName "kube-api-access-drdwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:52:14.416741 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:14.416722    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-maas-api-tls" pod "ae6fa740-5a10-4bd4-88cd-51f4006e79f5" (UID: "ae6fa740-5a10-4bd4-88cd-51f4006e79f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:52:14.519756 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:14.518034    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:14.519756 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:14.518074    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drdwj\" (UniqueName: \"kubernetes.io/projected/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-kube-api-access-drdwj\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:14.519756 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:14.518092    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae6fa740-5a10-4bd4-88cd-51f4006e79f5-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:14.582772 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:14.577877    4327 generic.go:356] "Generic (PLEG): container finished" podID="ae6fa740-5a10-4bd4-88cd-51f4006e79f5" containerID="374cd93c64cfa4778de289d6fe1b02af6f28b87f13cd6552f2bb443ecb82209a" exitCode=0
Oct 31 00:52:14.582772 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:14.578188    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s" event={"ID":"ae6fa740-5a10-4bd4-88cd-51f4006e79f5","Type":"ContainerDied","Data":"374cd93c64cfa4778de289d6fe1b02af6f28b87f13cd6552f2bb443ecb82209a"}
Oct 31 00:52:14.582772 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:14.578283    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s" event={"ID":"ae6fa740-5a10-4bd4-88cd-51f4006e79f5","Type":"ContainerDied","Data":"843eaef1a3a4d43aac4ffac9974edee9583c3dc14d9cf6adb1edb8ddf9b32d51"}
Oct 31 00:52:14.582772 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:14.578315    4327 scope.go:122] "RemoveContainer" containerID="374cd93c64cfa4778de289d6fe1b02af6f28b87f13cd6552f2bb443ecb82209a"
Oct 31 00:52:14.582772 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:14.578514    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s"
Oct 31 00:52:14.611401 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:14.611373    4327 scope.go:122] "RemoveContainer" containerID="374cd93c64cfa4778de289d6fe1b02af6f28b87f13cd6552f2bb443ecb82209a"
Oct 31 00:52:14.611781 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:52:14.611756    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"374cd93c64cfa4778de289d6fe1b02af6f28b87f13cd6552f2bb443ecb82209a\": container with ID starting with 374cd93c64cfa4778de289d6fe1b02af6f28b87f13cd6552f2bb443ecb82209a not found: ID does not exist" containerID="374cd93c64cfa4778de289d6fe1b02af6f28b87f13cd6552f2bb443ecb82209a"
Oct 31 00:52:14.611863 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:14.611792    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"374cd93c64cfa4778de289d6fe1b02af6f28b87f13cd6552f2bb443ecb82209a"} err="failed to get container status \"374cd93c64cfa4778de289d6fe1b02af6f28b87f13cd6552f2bb443ecb82209a\": rpc error: code = NotFound desc = could not find container \"374cd93c64cfa4778de289d6fe1b02af6f28b87f13cd6552f2bb443ecb82209a\": container with ID starting with 374cd93c64cfa4778de289d6fe1b02af6f28b87f13cd6552f2bb443ecb82209a not found: ID does not exist"
Oct 31 00:52:14.652793 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:14.652756    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s"]
Oct 31 00:52:14.658202 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:14.657955    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-j5w5s"]
Oct 31 00:52:15.296375 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:52:15.296337    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:52:15.502317 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:15.502283    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv"
Oct 31 00:52:15.923034 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:15.923003    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae6fa740-5a10-4bd4-88cd-51f4006e79f5" path="/var/lib/kubelet/pods/ae6fa740-5a10-4bd4-88cd-51f4006e79f5/volumes"
Oct 31 00:52:16.568976 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.568941    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6"]
Oct 31 00:52:16.569730 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.569714    4327 cpu_manager.go:414] "containerMap: removing container" podUID="14093068-2225-44ec-9ec6-9f85d5beeb4f" containerName="istio-proxy"
Oct 31 00:52:16.569781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.569732    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14093068-2225-44ec-9ec6-9f85d5beeb4f" containerName="istio-proxy"
Oct 31 00:52:16.569781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.569764    4327 cpu_manager.go:414] "containerMap: removing container" podUID="ae6fa740-5a10-4bd4-88cd-51f4006e79f5" containerName="maas-api"
Oct 31 00:52:16.569781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.569769    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae6fa740-5a10-4bd4-88cd-51f4006e79f5" containerName="maas-api"
Oct 31 00:52:16.569781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.569781    4327 cpu_manager.go:414] "containerMap: removing container" podUID="39a92afd-325e-4e33-9d0c-6b9fde7a49c3" containerName="payload-processing"
Oct 31 00:52:16.569910 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.569787    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39a92afd-325e-4e33-9d0c-6b9fde7a49c3" containerName="payload-processing"
Oct 31 00:52:16.569910 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.569803    4327 cpu_manager.go:414] "containerMap: removing container" podUID="14093068-2225-44ec-9ec6-9f85d5beeb4f" containerName="istio-proxy"
Oct 31 00:52:16.569910 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.569807    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14093068-2225-44ec-9ec6-9f85d5beeb4f" containerName="istio-proxy"
Oct 31 00:52:16.569910 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.569819    4327 cpu_manager.go:414] "containerMap: removing container" podUID="9751c672-027d-40c3-9da4-a18c4671f8fa" containerName="payload-pre-processing"
Oct 31 00:52:16.569910 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.569824    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9751c672-027d-40c3-9da4-a18c4671f8fa" containerName="payload-pre-processing"
Oct 31 00:52:16.570050 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.569946    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="14093068-2225-44ec-9ec6-9f85d5beeb4f" containerName="istio-proxy"
Oct 31 00:52:16.570050 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.569962    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae6fa740-5a10-4bd4-88cd-51f4006e79f5" containerName="maas-api"
Oct 31 00:52:16.570050 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.569970    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="39a92afd-325e-4e33-9d0c-6b9fde7a49c3" containerName="payload-processing"
Oct 31 00:52:16.570050 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.569979    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="aca0da87-ad0a-45cc-a022-d4946e7fab22" containerName="istio-proxy"
Oct 31 00:52:16.570050 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.569985    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="9751c672-027d-40c3-9da4-a18c4671f8fa" containerName="payload-pre-processing"
Oct 31 00:52:16.575411 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.575389    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6"
Oct 31 00:52:16.607513 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.607482    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-82d887-serving-cert\""
Oct 31 00:52:16.614113 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.614079    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6"]
Oct 31 00:52:16.651749 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.651715    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c45bbac-3d1d-459e-ab30-811a05b10ea0-maas-api-tls\") pod \"maas-api-e2e-api-b-82d887-d9469d9d9-7skb6\" (UID: \"6c45bbac-3d1d-459e-ab30-811a05b10ea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6"
Oct 31 00:52:16.651936 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.651759    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c45bbac-3d1d-459e-ab30-811a05b10ea0-openshift-service-ca\") pod \"maas-api-e2e-api-b-82d887-d9469d9d9-7skb6\" (UID: \"6c45bbac-3d1d-459e-ab30-811a05b10ea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6"
Oct 31 00:52:16.651936 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.651804    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x9dq\" (UniqueName: \"kubernetes.io/projected/6c45bbac-3d1d-459e-ab30-811a05b10ea0-kube-api-access-2x9dq\") pod \"maas-api-e2e-api-b-82d887-d9469d9d9-7skb6\" (UID: \"6c45bbac-3d1d-459e-ab30-811a05b10ea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6"
Oct 31 00:52:16.652024 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.651980    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c45bbac-3d1d-459e-ab30-811a05b10ea0-metrics-certs\") pod \"maas-api-e2e-api-b-82d887-d9469d9d9-7skb6\" (UID: \"6c45bbac-3d1d-459e-ab30-811a05b10ea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6"
Oct 31 00:52:16.774564 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.774520    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c45bbac-3d1d-459e-ab30-811a05b10ea0-maas-api-tls\") pod \"maas-api-e2e-api-b-82d887-d9469d9d9-7skb6\" (UID: \"6c45bbac-3d1d-459e-ab30-811a05b10ea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6"
Oct 31 00:52:16.788953 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.788903    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c45bbac-3d1d-459e-ab30-811a05b10ea0-openshift-service-ca\") pod \"maas-api-e2e-api-b-82d887-d9469d9d9-7skb6\" (UID: \"6c45bbac-3d1d-459e-ab30-811a05b10ea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6"
Oct 31 00:52:16.790093 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.790065    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x9dq\" (UniqueName: \"kubernetes.io/projected/6c45bbac-3d1d-459e-ab30-811a05b10ea0-kube-api-access-2x9dq\") pod \"maas-api-e2e-api-b-82d887-d9469d9d9-7skb6\" (UID: \"6c45bbac-3d1d-459e-ab30-811a05b10ea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6"
Oct 31 00:52:16.820610 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.813463    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c45bbac-3d1d-459e-ab30-811a05b10ea0-metrics-certs\") pod \"maas-api-e2e-api-b-82d887-d9469d9d9-7skb6\" (UID: \"6c45bbac-3d1d-459e-ab30-811a05b10ea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6"
Oct 31 00:52:16.820610 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.811280    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c45bbac-3d1d-459e-ab30-811a05b10ea0-maas-api-tls\") pod \"maas-api-e2e-api-b-82d887-d9469d9d9-7skb6\" (UID: \"6c45bbac-3d1d-459e-ab30-811a05b10ea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6"
Oct 31 00:52:16.820610 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.789943    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c45bbac-3d1d-459e-ab30-811a05b10ea0-openshift-service-ca\") pod \"maas-api-e2e-api-b-82d887-d9469d9d9-7skb6\" (UID: \"6c45bbac-3d1d-459e-ab30-811a05b10ea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6"
Oct 31 00:52:16.837405 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.830913    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x9dq\" (UniqueName: \"kubernetes.io/projected/6c45bbac-3d1d-459e-ab30-811a05b10ea0-kube-api-access-2x9dq\") pod \"maas-api-e2e-api-b-82d887-d9469d9d9-7skb6\" (UID: \"6c45bbac-3d1d-459e-ab30-811a05b10ea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6"
Oct 31 00:52:16.841741 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.841708    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c45bbac-3d1d-459e-ab30-811a05b10ea0-metrics-certs\") pod \"maas-api-e2e-api-b-82d887-d9469d9d9-7skb6\" (UID: \"6c45bbac-3d1d-459e-ab30-811a05b10ea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6"
Oct 31 00:52:16.902702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:16.897028    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6"
Oct 31 00:52:17.231352 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:17.231324    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6"]
Oct 31 00:52:17.233054 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:52:17.233020    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c45bbac_3d1d_459e_ab30_811a05b10ea0.slice/crio-5af4ce7598cffde9e4ad78f639427b2d690b770565e0e2cd8775deac879b37ac WatchSource:0}: Error finding container 5af4ce7598cffde9e4ad78f639427b2d690b770565e0e2cd8775deac879b37ac: Status 404 returned error can't find the container with id 5af4ce7598cffde9e4ad78f639427b2d690b770565e0e2cd8775deac879b37ac
Oct 31 00:52:17.658451 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:17.658415    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6" event={"ID":"6c45bbac-3d1d-459e-ab30-811a05b10ea0","Type":"ContainerStarted","Data":"5af4ce7598cffde9e4ad78f639427b2d690b770565e0e2cd8775deac879b37ac"}
Oct 31 00:52:18.668045 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:18.668008    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6" event={"ID":"6c45bbac-3d1d-459e-ab30-811a05b10ea0","Type":"ContainerStarted","Data":"451416a13b9d501b2f6975f5677a224b575f4c7b5011d02566ead81fe475c2ee"}
Oct 31 00:52:18.668462 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:18.668231    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6"
Oct 31 00:52:18.713945 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:18.713880    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6" podStartSLOduration=2.363316354 podStartE2EDuration="2.713864265s" totalImagesPullingTime="350.547911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:18.710362341 +0000 UTC m=+1998.932397910" watchObservedRunningTime="2026-10-31 00:52:18.713864265 +0000 UTC m=+1998.935899865"
Oct 31 00:52:19.979332 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:19.979296    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"]
Oct 31 00:52:24.681607 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:24.681575    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6"
Oct 31 00:52:35.054724 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:35.038227    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz" podUID="b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6" containerName="main" containerID="cri-o://f4d5e0f8eb34fbcb1d64a211c50cf91807c245020e0f2ddaf1a6a3e92f7d48a2" gracePeriod=45
Oct 31 00:52:35.183381 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:35.171879    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz" podUID="b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": dial tcp 10.133.0.100:8000: connect: connection refused"
Oct 31 00:52:35.656353 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:35.656316    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"
Oct 31 00:52:35.715959 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:35.715473    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-tls-certs\") pod \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\" (UID: \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\") "
Oct 31 00:52:35.739539 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:35.736957    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-tmp-dir\") pod \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\" (UID: \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\") "
Oct 31 00:52:35.739539 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:35.737199    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-model-cache\") pod \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\" (UID: \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\") "
Oct 31 00:52:35.739539 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:35.737319    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-kube-api-access-8cxhz\" (UniqueName: \"kubernetes.io/projected/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-kube-api-access-8cxhz\") pod \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\" (UID: \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\") "
Oct 31 00:52:35.739539 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:35.737417    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-home\" (UniqueName: \"kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-home\") pod \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\" (UID: \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\") "
Oct 31 00:52:35.739539 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:35.737488    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-dshm\") pod \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\" (UID: \"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6\") "
Oct 31 00:52:35.755413 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:35.745033    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-model-cache" pod "b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6" (UID: "b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:52:35.755413 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:35.745280    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-tmp-dir" pod "b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6" (UID: "b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:52:35.788051 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:35.785030    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-home" pod "b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6" (UID: "b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:52:35.847168 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:35.846784    4327 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-tmp-dir\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:35.847168 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:35.846822    4327 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-model-cache\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:35.847168 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:35.846839    4327 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-home\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:35.901946 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:35.901901    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-kube-api-access-8cxhz" pod "b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6" (UID: "b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6"). InnerVolumeSpecName "kube-api-access-8cxhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:52:35.902586 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:35.902552    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-dshm" pod "b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6" (UID: "b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:52:35.905830 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:35.905784    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-tls-certs" pod "b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6" (UID: "b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:52:35.954717 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:35.954121    4327 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-dshm\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:35.954717 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:35.954161    4327 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-tls-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:35.954717 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:35.954181    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cxhz\" (UniqueName: \"kubernetes.io/projected/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6-kube-api-access-8cxhz\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:52:35.974970 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:35.974926    4327 generic.go:356] "Generic (PLEG): container finished" podID="b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6" containerID="f4d5e0f8eb34fbcb1d64a211c50cf91807c245020e0f2ddaf1a6a3e92f7d48a2" exitCode=0
Oct 31 00:52:35.975219 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:35.975094    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"
Oct 31 00:52:35.981982 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:35.981938    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz" event={"ID":"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6","Type":"ContainerDied","Data":"f4d5e0f8eb34fbcb1d64a211c50cf91807c245020e0f2ddaf1a6a3e92f7d48a2"}
Oct 31 00:52:35.982156 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:35.982025    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz" event={"ID":"b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6","Type":"ContainerDied","Data":"da7c9d66addcdec157447a2115b6646414fa58c02ebe347b31e7648694fc4f47"}
Oct 31 00:52:35.982156 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:35.982053    4327 scope.go:122] "RemoveContainer" containerID="f4d5e0f8eb34fbcb1d64a211c50cf91807c245020e0f2ddaf1a6a3e92f7d48a2"
Oct 31 00:52:36.000941 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:36.000911    4327 scope.go:122] "RemoveContainer" containerID="f4d5e0f8eb34fbcb1d64a211c50cf91807c245020e0f2ddaf1a6a3e92f7d48a2"
Oct 31 00:52:36.001318 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:52:36.001295    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4d5e0f8eb34fbcb1d64a211c50cf91807c245020e0f2ddaf1a6a3e92f7d48a2\": container with ID starting with f4d5e0f8eb34fbcb1d64a211c50cf91807c245020e0f2ddaf1a6a3e92f7d48a2 not found: ID does not exist" containerID="f4d5e0f8eb34fbcb1d64a211c50cf91807c245020e0f2ddaf1a6a3e92f7d48a2"
Oct 31 00:52:36.001425 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:36.001334    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4d5e0f8eb34fbcb1d64a211c50cf91807c245020e0f2ddaf1a6a3e92f7d48a2"} err="failed to get container status \"f4d5e0f8eb34fbcb1d64a211c50cf91807c245020e0f2ddaf1a6a3e92f7d48a2\": rpc error: code = NotFound desc = could not find container \"f4d5e0f8eb34fbcb1d64a211c50cf91807c245020e0f2ddaf1a6a3e92f7d48a2\": container with ID starting with f4d5e0f8eb34fbcb1d64a211c50cf91807c245020e0f2ddaf1a6a3e92f7d48a2 not found: ID does not exist"
Oct 31 00:52:36.051336 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:36.051294    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"]
Oct 31 00:52:36.056840 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:36.056808    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-208e9f32/e2e-lifecycle-model-208e9f32-kserve-55554466bf-jnbqz"]
Oct 31 00:52:37.956808 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:37.956766    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6" path="/var/lib/kubelet/pods/b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6/volumes"
Oct 31 00:52:38.003725 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.001224    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"]
Oct 31 00:52:38.003725 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.002502    4327 cpu_manager.go:414] "containerMap: removing container" podUID="b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6" containerName="main"
Oct 31 00:52:38.003725 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.002524    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6" containerName="main"
Oct 31 00:52:38.003725 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.002761    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="14093068-2225-44ec-9ec6-9f85d5beeb4f" containerName="istio-proxy"
Oct 31 00:52:38.003725 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.002782    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1c60d59-fe16-4899-b8e4-e10fa5f7d9a6" containerName="main"
Oct 31 00:52:38.009528 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.009504    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.058383 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.046810    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c6d97d-openshift-default-dockercfg-65d9d\""
Oct 31 00:52:38.076433 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.076392    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"]
Oct 31 00:52:38.209735 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.209631    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79f228f5-1947-426c-9e4d-4f300f43511e-istio-podinfo\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.259229 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.259187    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-istio-envoy\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.276924 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.276879    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-credential-socket\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.304217 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.303852    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79f228f5-1947-426c-9e4d-4f300f43511e-istio-token\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.305477 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.304655    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg7cz\" (UniqueName: \"kubernetes.io/projected/79f228f5-1947-426c-9e4d-4f300f43511e-kube-api-access-sg7cz\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.307482 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.306667    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79f228f5-1947-426c-9e4d-4f300f43511e-istiod-ca-cert\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.332911 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.332872    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-istio-data\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.335395 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.335354    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-workload-certs\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.347726 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.339407    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-workload-socket\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.499377 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.469802    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-istio-data\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.499377 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.469953    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-workload-certs\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.499377 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.469989    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-workload-socket\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.499377 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.470136    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79f228f5-1947-426c-9e4d-4f300f43511e-istio-podinfo\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.499377 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.470197    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-istio-envoy\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.499377 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.470251    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-credential-socket\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.499377 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.470290    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79f228f5-1947-426c-9e4d-4f300f43511e-istio-token\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.499377 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.470326    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg7cz\" (UniqueName: \"kubernetes.io/projected/79f228f5-1947-426c-9e4d-4f300f43511e-kube-api-access-sg7cz\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.499377 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.470345    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-istio-data\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.499377 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.470371    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79f228f5-1947-426c-9e4d-4f300f43511e-istiod-ca-cert\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.499377 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.471214    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79f228f5-1947-426c-9e4d-4f300f43511e-istiod-ca-cert\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.499377 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.471286    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-credential-socket\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.499377 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.471868    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-workload-socket\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.499377 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.472109    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-workload-certs\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.499377 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.499063    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-istio-envoy\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.529728 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.529625    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg7cz\" (UniqueName: \"kubernetes.io/projected/79f228f5-1947-426c-9e4d-4f300f43511e-kube-api-access-sg7cz\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.532832 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.532768    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79f228f5-1947-426c-9e4d-4f300f43511e-istio-podinfo\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.546597 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.545470    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79f228f5-1947-426c-9e4d-4f300f43511e-istio-token\") pod \"e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") " pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:38.680221 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:38.680118    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:39.050181 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:39.050147    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"]
Oct 31 00:52:39.051277 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:52:39.051238    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f228f5_1947_426c_9e4d_4f300f43511e.slice/crio-661ef34188433bb35935d46a8dde691777eed1983e62b43b2bcde074408c86d6 WatchSource:0}: Error finding container 661ef34188433bb35935d46a8dde691777eed1983e62b43b2bcde074408c86d6: Status 404 returned error can't find the container with id 661ef34188433bb35935d46a8dde691777eed1983e62b43b2bcde074408c86d6
Oct 31 00:52:39.054664 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:39.054620    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:52:39.054816 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:39.054789    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:52:39.054870 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:39.054853    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:52:40.018170 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:40.018127    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d" event={"ID":"79f228f5-1947-426c-9e4d-4f300f43511e","Type":"ContainerStarted","Data":"6a50b7d905e033190187c39065ee1df2811ab7c476e902c359e8fdedd4b5b8ea"}
Oct 31 00:52:40.018353 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:40.018199    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d" event={"ID":"79f228f5-1947-426c-9e4d-4f300f43511e","Type":"ContainerStarted","Data":"661ef34188433bb35935d46a8dde691777eed1983e62b43b2bcde074408c86d6"}
Oct 31 00:52:40.058888 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:40.058826    4327 patch_prober.go:29] interesting pod/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:52:40.059595 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:40.059559    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv" podUID="47727a40-911f-4d8d-8fd6-f19ada8e4ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:52:40.106746 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:40.098816    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d" podStartSLOduration=3.098796872 podStartE2EDuration="3.098796872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:40.096483558 +0000 UTC m=+2020.318519150" watchObservedRunningTime="2026-10-31 00:52:40.098796872 +0000 UTC m=+2020.320832415"
Oct 31 00:52:40.680953 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:40.680909    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:40.688278 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:40.688250    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:41.029066 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:41.028980    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:41.030166 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:41.030142    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:52:42.039394 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:42.039355    4327 generic.go:356] "Generic (PLEG): container finished" podID="47727a40-911f-4d8d-8fd6-f19ada8e4ea0" containerID="8514da30c40f9a548d39bd46eb762a911457d8b78e6f4b792b4a77dfa98caa47" exitCode=0
Oct 31 00:52:42.040210 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:42.039425    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv" event={"ID":"47727a40-911f-4d8d-8fd6-f19ada8e4ea0","Type":"ContainerDied","Data":"8514da30c40f9a548d39bd46eb762a911457d8b78e6f4b792b4a77dfa98caa47"}
Oct 31 00:52:42.040210 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:42.040009    4327 scope.go:122] "RemoveContainer" containerID="8514da30c40f9a548d39bd46eb762a911457d8b78e6f4b792b4a77dfa98caa47"
Oct 31 00:52:42.041890 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:42.041871    4327 generic.go:356] "Generic (PLEG): container finished" podID="46ed7530-493b-48d4-ac03-a4d9179504ff" containerID="aa239a74c88519d1ce63bd77b96863351fced16f790a1140af92e1b40d72ea9f" exitCode=0
Oct 31 00:52:42.042277 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:42.042021    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt" event={"ID":"46ed7530-493b-48d4-ac03-a4d9179504ff","Type":"ContainerDied","Data":"aa239a74c88519d1ce63bd77b96863351fced16f790a1140af92e1b40d72ea9f"}
Oct 31 00:52:42.042363 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:42.042311    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:52:42.042428 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:42.042385    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:52:42.042428 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:42.042417    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:52:42.042662 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:42.042645    4327 scope.go:122] "RemoveContainer" containerID="aa239a74c88519d1ce63bd77b96863351fced16f790a1140af92e1b40d72ea9f"
Oct 31 00:52:42.044349 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:42.044309    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:52:42.044459 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:42.044387    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:52:42.044459 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:42.044438    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:52:43.090166 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:43.090123    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv" event={"ID":"47727a40-911f-4d8d-8fd6-f19ada8e4ea0","Type":"ContainerStarted","Data":"b94f41e5ac21207e05812b0614472387e51061e4bd6c94b34a6a2e58c681779f"}
Oct 31 00:52:43.092621 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:43.092592    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt" event={"ID":"46ed7530-493b-48d4-ac03-a4d9179504ff","Type":"ContainerStarted","Data":"f158cf25842ee8d3c70bcaa2740451a81bf0475e18550ed9c8c4ee2c936bfdbf"}
Oct 31 00:52:43.099355 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:43.099332    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:52:43.100517 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:43.100487    4327 patch_prober.go:29] interesting pod/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 00:52:43.100599 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:43.100553    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt" podUID="46ed7530-493b-48d4-ac03-a4d9179504ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 00:52:43.784538 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:43.784495    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:52:43.784942 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:43.784916    4327 patch_prober.go:29] interesting pod/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 31 00:52:43.785050 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:43.784988    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv" podUID="47727a40-911f-4d8d-8fd6-f19ada8e4ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 31 00:52:44.099406 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:44.099308    4327 patch_prober.go:29] interesting pod/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 00:52:44.099406 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:44.099373    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt" podUID="46ed7530-493b-48d4-ac03-a4d9179504ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 00:52:44.598190 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:44.598154    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:52:44.673752 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:44.637786    4327 patch_prober.go:29] interesting pod/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:52:44.673752 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:44.637892    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht" podUID="e30c1c23-92c5-47b6-a250-3519dbfae94b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:52:44.785253 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:44.785168    4327 patch_prober.go:29] interesting pod/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 31 00:52:44.785253 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:44.785226    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv" podUID="47727a40-911f-4d8d-8fd6-f19ada8e4ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 31 00:52:44.873143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:44.873053    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9"]
Oct 31 00:52:44.880092 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:44.879828    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9"]
Oct 31 00:52:44.880092 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:44.879920    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9"
Oct 31 00:52:44.896317 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:44.896233    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c6d97d\""
Oct 31 00:52:44.896746 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:44.896713    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c6d97d-dockercfg-r64qb\""
Oct 31 00:52:44.986706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:44.985917    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67570d88-4047-46e3-9c60-3522e8c59493-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9\" (UID: \"67570d88-4047-46e3-9c60-3522e8c59493\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9"
Oct 31 00:52:44.986706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:44.986080    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbtzz\" (UniqueName: \"kubernetes.io/projected/67570d88-4047-46e3-9c60-3522e8c59493-kube-api-access-qbtzz\") pod \"payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9\" (UID: \"67570d88-4047-46e3-9c60-3522e8c59493\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9"
Oct 31 00:52:45.104979 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:45.095653    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbtzz\" (UniqueName: \"kubernetes.io/projected/67570d88-4047-46e3-9c60-3522e8c59493-kube-api-access-qbtzz\") pod \"payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9\" (UID: \"67570d88-4047-46e3-9c60-3522e8c59493\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9"
Oct 31 00:52:45.104979 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:45.095977    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67570d88-4047-46e3-9c60-3522e8c59493-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9\" (UID: \"67570d88-4047-46e3-9c60-3522e8c59493\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9"
Oct 31 00:52:45.104979 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:45.097042    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67570d88-4047-46e3-9c60-3522e8c59493-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9\" (UID: \"67570d88-4047-46e3-9c60-3522e8c59493\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9"
Oct 31 00:52:45.104979 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:45.099923    4327 patch_prober.go:29] interesting pod/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 00:52:45.104979 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:45.099970    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt" podUID="46ed7530-493b-48d4-ac03-a4d9179504ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 00:52:45.380163 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:45.379410    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbtzz\" (UniqueName: \"kubernetes.io/projected/67570d88-4047-46e3-9c60-3522e8c59493-kube-api-access-qbtzz\") pod \"payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9\" (UID: \"67570d88-4047-46e3-9c60-3522e8c59493\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9"
Oct 31 00:52:45.399699 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:45.399638    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd"]
Oct 31 00:52:45.410715 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:45.410663    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd"]
Oct 31 00:52:45.410906 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:45.410799    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd"
Oct 31 00:52:45.479906 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:52:45.479868    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:52:45.488252 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:45.488225    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlzgd\" (UniqueName: \"kubernetes.io/projected/d628dce5-6346-46cc-a356-ad526ffd0b96-kube-api-access-tlzgd\") pod \"payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd\" (UID: \"d628dce5-6346-46cc-a356-ad526ffd0b96\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd"
Oct 31 00:52:45.488415 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:45.488283    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d628dce5-6346-46cc-a356-ad526ffd0b96-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd\" (UID: \"d628dce5-6346-46cc-a356-ad526ffd0b96\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd"
Oct 31 00:52:45.496992 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:45.496968    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9"
Oct 31 00:52:45.589459 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:45.589409    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d628dce5-6346-46cc-a356-ad526ffd0b96-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd\" (UID: \"d628dce5-6346-46cc-a356-ad526ffd0b96\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd"
Oct 31 00:52:45.589758 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:45.589733    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlzgd\" (UniqueName: \"kubernetes.io/projected/d628dce5-6346-46cc-a356-ad526ffd0b96-kube-api-access-tlzgd\") pod \"payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd\" (UID: \"d628dce5-6346-46cc-a356-ad526ffd0b96\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd"
Oct 31 00:52:45.590467 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:45.590446    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d628dce5-6346-46cc-a356-ad526ffd0b96-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd\" (UID: \"d628dce5-6346-46cc-a356-ad526ffd0b96\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd"
Oct 31 00:52:45.657198 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:45.614840    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlzgd\" (UniqueName: \"kubernetes.io/projected/d628dce5-6346-46cc-a356-ad526ffd0b96-kube-api-access-tlzgd\") pod \"payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd\" (UID: \"d628dce5-6346-46cc-a356-ad526ffd0b96\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd"
Oct 31 00:52:45.783011 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:45.782972    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd"
Oct 31 00:52:45.785723 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:45.785345    4327 patch_prober.go:29] interesting pod/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 31 00:52:45.785723 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:45.785405    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv" podUID="47727a40-911f-4d8d-8fd6-f19ada8e4ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 31 00:52:46.049103 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:52:46.049063    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67570d88_4047_46e3_9c60_3522e8c59493.slice/crio-ac2a905f0d575930b2337d4798a06e21ac664e811ce35294ff86dd3c21af68e9 WatchSource:0}: Error finding container ac2a905f0d575930b2337d4798a06e21ac664e811ce35294ff86dd3c21af68e9: Status 404 returned error can't find the container with id ac2a905f0d575930b2337d4798a06e21ac664e811ce35294ff86dd3c21af68e9
Oct 31 00:52:46.088336 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:46.088298    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9"]
Oct 31 00:52:46.156819 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:46.152197    4327 patch_prober.go:29] interesting pod/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 00:52:46.156819 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:46.152258    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt" podUID="46ed7530-493b-48d4-ac03-a4d9179504ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 00:52:46.405762 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:46.405724    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9" event={"ID":"67570d88-4047-46e3-9c60-3522e8c59493","Type":"ContainerStarted","Data":"ac2a905f0d575930b2337d4798a06e21ac664e811ce35294ff86dd3c21af68e9"}
Oct 31 00:52:46.785708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:46.785105    4327 patch_prober.go:29] interesting pod/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 31 00:52:46.785708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:46.785182    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv" podUID="47727a40-911f-4d8d-8fd6-f19ada8e4ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 31 00:52:46.801605 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:46.801474    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd"]
Oct 31 00:52:47.100293 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:47.100259    4327 patch_prober.go:29] interesting pod/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 00:52:47.100449 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:47.100325    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt" podUID="46ed7530-493b-48d4-ac03-a4d9179504ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 00:52:47.464774 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:47.438565    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9" event={"ID":"67570d88-4047-46e3-9c60-3522e8c59493","Type":"ContainerStarted","Data":"a54e5003fd0d96d98aca740d43e27a7c303bd014adf735ec1b9e5370510b72bc"}
Oct 31 00:52:47.464774 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:47.441467    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9"
Oct 31 00:52:47.479040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:47.470524    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd" event={"ID":"d628dce5-6346-46cc-a356-ad526ffd0b96","Type":"ContainerStarted","Data":"d77a8c61c24b16dbc4269489c7d97b64eacdff26c7c1922c2043e0633389359c"}
Oct 31 00:52:47.479040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:47.470594    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd" event={"ID":"d628dce5-6346-46cc-a356-ad526ffd0b96","Type":"ContainerStarted","Data":"312ab59c37e3a931ee09cf11cedf3a064896aae89ebeaae7eb41da7addbfdac6"}
Oct 31 00:52:47.479040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:47.471555    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd"
Oct 31 00:52:47.488742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:47.487761    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9" podStartSLOduration=3.487745767 podStartE2EDuration="3.487745767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:47.486777143 +0000 UTC m=+2027.708812686" watchObservedRunningTime="2026-10-31 00:52:47.487745767 +0000 UTC m=+2027.709781309"
Oct 31 00:52:47.797169 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:47.797074    4327 patch_prober.go:29] interesting pod/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 31 00:52:47.797448 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:47.797415    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv" podUID="47727a40-911f-4d8d-8fd6-f19ada8e4ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 31 00:52:48.099266 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:48.099178    4327 patch_prober.go:29] interesting pod/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 00:52:48.099266 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:48.099237    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt" podUID="46ed7530-493b-48d4-ac03-a4d9179504ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 00:52:48.497305 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:48.497264    4327 generic.go:356] "Generic (PLEG): container finished" podID="e30c1c23-92c5-47b6-a250-3519dbfae94b" containerID="ab5032d092a1c46a9a13905c1fd3dd5df965157cba66865ffc90ace375d5a265" exitCode=0
Oct 31 00:52:48.498335 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:48.498305    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht" event={"ID":"e30c1c23-92c5-47b6-a250-3519dbfae94b","Type":"ContainerDied","Data":"ab5032d092a1c46a9a13905c1fd3dd5df965157cba66865ffc90ace375d5a265"}
Oct 31 00:52:48.498823 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:48.498799    4327 scope.go:122] "RemoveContainer" containerID="ab5032d092a1c46a9a13905c1fd3dd5df965157cba66865ffc90ace375d5a265"
Oct 31 00:52:48.501705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:48.501650    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:52:48.502301 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:48.502268    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:52:48.502397 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:48.502336    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:52:48.680933 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:48.680868    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd" podStartSLOduration=4.6808457180000005 podStartE2EDuration="4.680845718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:47.550024125 +0000 UTC m=+2027.772059679" watchObservedRunningTime="2026-10-31 00:52:48.680845718 +0000 UTC m=+2028.902881261"
Oct 31 00:52:48.693275 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:48.693253    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt"]
Oct 31 00:52:48.714738 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:48.714714    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt"
Oct 31 00:52:48.811315 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:48.811279    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-bdb61f-dockercfg-zggtj\""
Oct 31 00:52:48.877711 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:48.850259    4327 patch_prober.go:29] interesting pod/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 31 00:52:48.877711 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:48.850319    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv" podUID="47727a40-911f-4d8d-8fd6-f19ada8e4ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 31 00:52:48.877711 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:48.851746    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt"]
Oct 31 00:52:48.877711 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:48.861122    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-bdb61f\""
Oct 31 00:52:48.901664 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:48.901623    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31633996-ae18-4e91-bc33-f604afbefade-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt\" (UID: \"31633996-ae18-4e91-bc33-f604afbefade\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt"
Oct 31 00:52:48.902022 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:48.902001    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbstv\" (UniqueName: \"kubernetes.io/projected/31633996-ae18-4e91-bc33-f604afbefade-kube-api-access-qbstv\") pod \"payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt\" (UID: \"31633996-ae18-4e91-bc33-f604afbefade\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt"
Oct 31 00:52:49.005473 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:49.003796    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbstv\" (UniqueName: \"kubernetes.io/projected/31633996-ae18-4e91-bc33-f604afbefade-kube-api-access-qbstv\") pod \"payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt\" (UID: \"31633996-ae18-4e91-bc33-f604afbefade\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt"
Oct 31 00:52:49.005473 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:49.004243    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31633996-ae18-4e91-bc33-f604afbefade-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt\" (UID: \"31633996-ae18-4e91-bc33-f604afbefade\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt"
Oct 31 00:52:49.005473 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:49.005221    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31633996-ae18-4e91-bc33-f604afbefade-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt\" (UID: \"31633996-ae18-4e91-bc33-f604afbefade\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt"
Oct 31 00:52:49.144707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:49.126672    4327 patch_prober.go:29] interesting pod/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 00:52:49.144707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:49.126759    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt" podUID="46ed7530-493b-48d4-ac03-a4d9179504ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 00:52:49.179232 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:49.155421    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbstv\" (UniqueName: \"kubernetes.io/projected/31633996-ae18-4e91-bc33-f604afbefade-kube-api-access-qbstv\") pod \"payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt\" (UID: \"31633996-ae18-4e91-bc33-f604afbefade\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt"
Oct 31 00:52:49.179232 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:49.169859    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt"
Oct 31 00:52:49.525885 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:49.525539    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht" event={"ID":"e30c1c23-92c5-47b6-a250-3519dbfae94b","Type":"ContainerStarted","Data":"7a7aec5ea85a41c0f310a46441a132ed668b7b400d2679fc50f3cbd031e5cf49"}
Oct 31 00:52:49.667628 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:49.667587    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt"]
Oct 31 00:52:49.865036 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:49.862032    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:52:49.865036 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:49.863691    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:52:49.882192 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:49.881104    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 00:52:50.106865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:50.106825    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:52:50.108078 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:50.108054    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:52:50.269876 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:50.269844    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:52:50.271133 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:50.271097    4327 patch_prober.go:29] interesting pod/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 00:52:50.271279 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:50.271166    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht" podUID="e30c1c23-92c5-47b6-a250-3519dbfae94b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 00:52:50.552207 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:50.552124    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt" event={"ID":"31633996-ae18-4e91-bc33-f604afbefade","Type":"ContainerStarted","Data":"018eca14a58d0a6aeed78d5c07c5855bf93f4d54e37b398149c861381829fb77"}
Oct 31 00:52:50.552207 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:50.552181    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt" event={"ID":"31633996-ae18-4e91-bc33-f604afbefade","Type":"ContainerStarted","Data":"e80fb03766866cb893ab78e319ad48742f982ef610a330e2e22edca380d7743d"}
Oct 31 00:52:50.552771 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:50.552214    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt"
Oct 31 00:52:51.270878 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:51.270846    4327 patch_prober.go:29] interesting pod/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 00:52:51.271058 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:51.270894    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht" podUID="e30c1c23-92c5-47b6-a250-3519dbfae94b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 00:52:52.270931 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:52.270894    4327 patch_prober.go:29] interesting pod/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 00:52:52.271350 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:52.270947    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht" podUID="e30c1c23-92c5-47b6-a250-3519dbfae94b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 00:52:53.270016 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:53.269979    4327 patch_prober.go:29] interesting pod/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 00:52:53.270197 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:53.270036    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht" podUID="e30c1c23-92c5-47b6-a250-3519dbfae94b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 00:52:54.270212 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:54.270176    4327 patch_prober.go:29] interesting pod/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 00:52:54.270638 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:54.270229    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht" podUID="e30c1c23-92c5-47b6-a250-3519dbfae94b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 00:52:54.954577 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:54.954519    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt" podStartSLOduration=6.954504546 podStartE2EDuration="6.954504546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:50.613935634 +0000 UTC m=+2030.835971177" watchObservedRunningTime="2026-10-31 00:52:54.954504546 +0000 UTC m=+2035.176540089"
Oct 31 00:52:54.954781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:54.954769    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t"]
Oct 31 00:52:54.960017 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:54.959995    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t"
Oct 31 00:52:54.980420 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:54.980391    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t"]
Oct 31 00:52:54.986897 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:54.986873    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-609843\""
Oct 31 00:52:54.986897 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:54.986883    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-609843-dockercfg-5gx8h\""
Oct 31 00:52:55.058713 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:55.056457    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhtpv\" (UniqueName: \"kubernetes.io/projected/2108960e-cdf0-40c8-a646-dbdcecc96099-kube-api-access-bhtpv\") pod \"payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t\" (UID: \"2108960e-cdf0-40c8-a646-dbdcecc96099\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t"
Oct 31 00:52:55.058713 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:55.056638    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2108960e-cdf0-40c8-a646-dbdcecc96099-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t\" (UID: \"2108960e-cdf0-40c8-a646-dbdcecc96099\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t"
Oct 31 00:52:55.196741 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:55.196352    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhtpv\" (UniqueName: \"kubernetes.io/projected/2108960e-cdf0-40c8-a646-dbdcecc96099-kube-api-access-bhtpv\") pod \"payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t\" (UID: \"2108960e-cdf0-40c8-a646-dbdcecc96099\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t"
Oct 31 00:52:55.196741 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:55.196483    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2108960e-cdf0-40c8-a646-dbdcecc96099-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t\" (UID: \"2108960e-cdf0-40c8-a646-dbdcecc96099\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t"
Oct 31 00:52:55.269771 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:55.266789    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2108960e-cdf0-40c8-a646-dbdcecc96099-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t\" (UID: \"2108960e-cdf0-40c8-a646-dbdcecc96099\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t"
Oct 31 00:52:55.309132 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:55.309093    4327 patch_prober.go:29] interesting pod/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 00:52:55.317813 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:55.317765    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht" podUID="e30c1c23-92c5-47b6-a250-3519dbfae94b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 00:52:55.383703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:55.380754    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-609843-b8d75c975-frvqn"]
Oct 31 00:52:55.482582 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:55.482547    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-609843-b8d75c975-frvqn"]
Oct 31 00:52:55.482928 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:55.482909    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-609843-b8d75c975-frvqn"
Oct 31 00:52:55.578505 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:55.578454    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhtpv\" (UniqueName: \"kubernetes.io/projected/2108960e-cdf0-40c8-a646-dbdcecc96099-kube-api-access-bhtpv\") pod \"payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t\" (UID: \"2108960e-cdf0-40c8-a646-dbdcecc96099\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t"
Oct 31 00:52:55.579231 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:55.579203    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t"
Oct 31 00:52:55.639131 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:55.639066    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whm45\" (UniqueName: \"kubernetes.io/projected/80b7e0a0-98d7-4176-9657-dcd37514545b-kube-api-access-whm45\") pod \"payload-processing-e2e-inf-b-609843-b8d75c975-frvqn\" (UID: \"80b7e0a0-98d7-4176-9657-dcd37514545b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-609843-b8d75c975-frvqn"
Oct 31 00:52:55.639442 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:55.639357    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b7e0a0-98d7-4176-9657-dcd37514545b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-609843-b8d75c975-frvqn\" (UID: \"80b7e0a0-98d7-4176-9657-dcd37514545b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-609843-b8d75c975-frvqn"
Oct 31 00:52:55.740917 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:55.740876    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b7e0a0-98d7-4176-9657-dcd37514545b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-609843-b8d75c975-frvqn\" (UID: \"80b7e0a0-98d7-4176-9657-dcd37514545b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-609843-b8d75c975-frvqn"
Oct 31 00:52:55.741179 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:55.741161    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whm45\" (UniqueName: \"kubernetes.io/projected/80b7e0a0-98d7-4176-9657-dcd37514545b-kube-api-access-whm45\") pod \"payload-processing-e2e-inf-b-609843-b8d75c975-frvqn\" (UID: \"80b7e0a0-98d7-4176-9657-dcd37514545b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-609843-b8d75c975-frvqn"
Oct 31 00:52:55.741739 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:55.741712    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b7e0a0-98d7-4176-9657-dcd37514545b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-609843-b8d75c975-frvqn\" (UID: \"80b7e0a0-98d7-4176-9657-dcd37514545b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-609843-b8d75c975-frvqn"
Oct 31 00:52:55.765091 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:55.765066    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whm45\" (UniqueName: \"kubernetes.io/projected/80b7e0a0-98d7-4176-9657-dcd37514545b-kube-api-access-whm45\") pod \"payload-processing-e2e-inf-b-609843-b8d75c975-frvqn\" (UID: \"80b7e0a0-98d7-4176-9657-dcd37514545b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-609843-b8d75c975-frvqn"
Oct 31 00:52:55.896280 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:55.896241    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-609843-b8d75c975-frvqn"
Oct 31 00:52:55.929034 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:55.929006    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t"]
Oct 31 00:52:55.929458 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:52:55.929427    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2108960e_cdf0_40c8_a646_dbdcecc96099.slice/crio-35465f255f504aa93824a44b5cddb82cb86ff2ee3f2faf80f61be0f29070f389 WatchSource:0}: Error finding container 35465f255f504aa93824a44b5cddb82cb86ff2ee3f2faf80f61be0f29070f389: Status 404 returned error can't find the container with id 35465f255f504aa93824a44b5cddb82cb86ff2ee3f2faf80f61be0f29070f389
Oct 31 00:52:56.234093 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:56.234000    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-609843-b8d75c975-frvqn"]
Oct 31 00:52:56.261368 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:52:56.261305    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b7e0a0_98d7_4176_9657_dcd37514545b.slice/crio-2d2c3cb23b0bedd9246d57b2ee95c5afd945a16d49120a41d275878bba717354 WatchSource:0}: Error finding container 2d2c3cb23b0bedd9246d57b2ee95c5afd945a16d49120a41d275878bba717354: Status 404 returned error can't find the container with id 2d2c3cb23b0bedd9246d57b2ee95c5afd945a16d49120a41d275878bba717354
Oct 31 00:52:56.277406 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:56.277368    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:52:56.278366 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:56.278336    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:52:56.279301 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:56.279265    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 00:52:56.619036 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:56.618993    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-609843-b8d75c975-frvqn" event={"ID":"80b7e0a0-98d7-4176-9657-dcd37514545b","Type":"ContainerStarted","Data":"f9f4eb5ebfc640fb770e1006ae2ebad7bbf53a96e779f380e831954e4104c986"}
Oct 31 00:52:56.619403 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:56.619097    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-609843-b8d75c975-frvqn" event={"ID":"80b7e0a0-98d7-4176-9657-dcd37514545b","Type":"ContainerStarted","Data":"2d2c3cb23b0bedd9246d57b2ee95c5afd945a16d49120a41d275878bba717354"}
Oct 31 00:52:56.619727 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:56.619709    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-609843-b8d75c975-frvqn"
Oct 31 00:52:56.620758 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:56.620734    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t" event={"ID":"2108960e-cdf0-40c8-a646-dbdcecc96099","Type":"ContainerStarted","Data":"97ac57aa39f8bb899bb03c06c26065c94b54f3d2ebbcad52a2dc30f3e2327ace"}
Oct 31 00:52:56.620841 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:56.620782    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t" event={"ID":"2108960e-cdf0-40c8-a646-dbdcecc96099","Type":"ContainerStarted","Data":"35465f255f504aa93824a44b5cddb82cb86ff2ee3f2faf80f61be0f29070f389"}
Oct 31 00:52:56.690536 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:56.690484    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-609843-b8d75c975-frvqn" podStartSLOduration=1.690469304 podStartE2EDuration="1.690469304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:56.68621883 +0000 UTC m=+2036.908254372" watchObservedRunningTime="2026-10-31 00:52:56.690469304 +0000 UTC m=+2036.912504845"
Oct 31 00:52:56.742122 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:56.742061    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t" podStartSLOduration=2.742046336 podStartE2EDuration="2.742046336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:56.737040049 +0000 UTC m=+2036.959075582" watchObservedRunningTime="2026-10-31 00:52:56.742046336 +0000 UTC m=+2036.964081878"
Oct 31 00:52:57.666919 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:57.666883    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t"
Oct 31 00:52:59.529440 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:59.529388    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9"
Oct 31 00:52:59.531254 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:52:59.531232    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd"
Oct 31 00:53:01.563391 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:01.563357    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt"
Oct 31 00:53:04.918071 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:04.918034    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"]
Oct 31 00:53:04.918668 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:04.918273    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms" podUID="5ff44805-0009-41dc-9415-cc05817837ea" containerName="istio-proxy" containerID="cri-o://702a622173a4630c5bd54e3e19145cf6f1601c7bef627fed08d39d89452fdaa5" gracePeriod=30
Oct 31 00:53:07.673357 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:07.673329    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-609843-b8d75c975-frvqn"
Oct 31 00:53:08.685920 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:08.685889    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t"
Oct 31 00:53:09.044098 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:09.044017    4327 patch_prober.go:29] interesting pod/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:53:09.044259 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:09.044097    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms" podUID="5ff44805-0009-41dc-9415-cc05817837ea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:53:10.223074 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.223047    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:53:10.329026 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.328993    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ff44805-0009-41dc-9415-cc05817837ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ff44805-0009-41dc-9415-cc05817837ea-istio-podinfo\") pod \"5ff44805-0009-41dc-9415-cc05817837ea\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") "
Oct 31 00:53:10.329230 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.329063    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-workload-socket\") pod \"5ff44805-0009-41dc-9415-cc05817837ea\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") "
Oct 31 00:53:10.329230 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.329147    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-istio-data\") pod \"5ff44805-0009-41dc-9415-cc05817837ea\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") "
Oct 31 00:53:10.329346 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.329288    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ff44805-0009-41dc-9415-cc05817837ea-istio-token\" (UniqueName: \"kubernetes.io/projected/5ff44805-0009-41dc-9415-cc05817837ea-istio-token\") pod \"5ff44805-0009-41dc-9415-cc05817837ea\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") "
Oct 31 00:53:10.329476 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.329458    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-istio-envoy\") pod \"5ff44805-0009-41dc-9415-cc05817837ea\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") "
Oct 31 00:53:10.329669 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.329639    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-istio-data" pod "5ff44805-0009-41dc-9415-cc05817837ea" (UID: "5ff44805-0009-41dc-9415-cc05817837ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:10.329792 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.329659    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-credential-socket\") pod \"5ff44805-0009-41dc-9415-cc05817837ea\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") "
Oct 31 00:53:10.329792 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.329652    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-workload-socket" pod "5ff44805-0009-41dc-9415-cc05817837ea" (UID: "5ff44805-0009-41dc-9415-cc05817837ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:10.329792 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.329735    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-workload-certs\") pod \"5ff44805-0009-41dc-9415-cc05817837ea\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") "
Oct 31 00:53:10.329792 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.329770    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ff44805-0009-41dc-9415-cc05817837ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ff44805-0009-41dc-9415-cc05817837ea-istiod-ca-cert\") pod \"5ff44805-0009-41dc-9415-cc05817837ea\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") "
Oct 31 00:53:10.330012 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.329828    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ff44805-0009-41dc-9415-cc05817837ea-kube-api-access-xgmlv\" (UniqueName: \"kubernetes.io/projected/5ff44805-0009-41dc-9415-cc05817837ea-kube-api-access-xgmlv\") pod \"5ff44805-0009-41dc-9415-cc05817837ea\" (UID: \"5ff44805-0009-41dc-9415-cc05817837ea\") "
Oct 31 00:53:10.330012 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.329956    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-credential-socket" pod "5ff44805-0009-41dc-9415-cc05817837ea" (UID: "5ff44805-0009-41dc-9415-cc05817837ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:10.330374 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.330172    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-workload-certs" pod "5ff44805-0009-41dc-9415-cc05817837ea" (UID: "5ff44805-0009-41dc-9415-cc05817837ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:10.330374 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.330285    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff44805-0009-41dc-9415-cc05817837ea-istiod-ca-cert" pod "5ff44805-0009-41dc-9415-cc05817837ea" (UID: "5ff44805-0009-41dc-9415-cc05817837ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:53:10.330929 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.330838    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:53:10.330929 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.330865    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:53:10.330929 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.330879    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:53:10.330929 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.330892    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:53:10.330929 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.330906    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ff44805-0009-41dc-9415-cc05817837ea-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:53:10.332748 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.332725    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-istio-envoy" pod "5ff44805-0009-41dc-9415-cc05817837ea" (UID: "5ff44805-0009-41dc-9415-cc05817837ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:10.332961 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.332940    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff44805-0009-41dc-9415-cc05817837ea-kube-api-access-xgmlv" pod "5ff44805-0009-41dc-9415-cc05817837ea" (UID: "5ff44805-0009-41dc-9415-cc05817837ea"). InnerVolumeSpecName "kube-api-access-xgmlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:53:10.333061 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.332975    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff44805-0009-41dc-9415-cc05817837ea-istio-token" pod "5ff44805-0009-41dc-9415-cc05817837ea" (UID: "5ff44805-0009-41dc-9415-cc05817837ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:53:10.333131 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.333083    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ff44805-0009-41dc-9415-cc05817837ea-istio-podinfo" pod "5ff44805-0009-41dc-9415-cc05817837ea" (UID: "5ff44805-0009-41dc-9415-cc05817837ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:53:10.432000 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.431970    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ff44805-0009-41dc-9415-cc05817837ea-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:53:10.432000 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.431997    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ff44805-0009-41dc-9415-cc05817837ea-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:53:10.432000 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.432006    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ff44805-0009-41dc-9415-cc05817837ea-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:53:10.432244 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.432014    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgmlv\" (UniqueName: \"kubernetes.io/projected/5ff44805-0009-41dc-9415-cc05817837ea-kube-api-access-xgmlv\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:53:10.629306 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.629269    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf"]
Oct 31 00:53:10.668623 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.668585    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54"]
Oct 31 00:53:10.871293 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.871255    4327 generic.go:356] "Generic (PLEG): container finished" podID="5ff44805-0009-41dc-9415-cc05817837ea" containerID="702a622173a4630c5bd54e3e19145cf6f1601c7bef627fed08d39d89452fdaa5" exitCode=0
Oct 31 00:53:10.871493 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.871320    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"
Oct 31 00:53:10.871493 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.871369    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms" event={"ID":"5ff44805-0009-41dc-9415-cc05817837ea","Type":"ContainerDied","Data":"702a622173a4630c5bd54e3e19145cf6f1601c7bef627fed08d39d89452fdaa5"}
Oct 31 00:53:10.871493 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.871424    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms" event={"ID":"5ff44805-0009-41dc-9415-cc05817837ea","Type":"ContainerDied","Data":"b7c73fe105a67b44c49846f73e900702a010cd72c429d8f4df73c38372b1ce03"}
Oct 31 00:53:10.871493 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.871446    4327 scope.go:122] "RemoveContainer" containerID="702a622173a4630c5bd54e3e19145cf6f1601c7bef627fed08d39d89452fdaa5"
Oct 31 00:53:10.917623 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.916084    4327 scope.go:122] "RemoveContainer" containerID="702a622173a4630c5bd54e3e19145cf6f1601c7bef627fed08d39d89452fdaa5"
Oct 31 00:53:10.918373 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:53:10.918263    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"702a622173a4630c5bd54e3e19145cf6f1601c7bef627fed08d39d89452fdaa5\": container with ID starting with 702a622173a4630c5bd54e3e19145cf6f1601c7bef627fed08d39d89452fdaa5 not found: ID does not exist" containerID="702a622173a4630c5bd54e3e19145cf6f1601c7bef627fed08d39d89452fdaa5"
Oct 31 00:53:10.918373 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.918302    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"702a622173a4630c5bd54e3e19145cf6f1601c7bef627fed08d39d89452fdaa5"} err="failed to get container status \"702a622173a4630c5bd54e3e19145cf6f1601c7bef627fed08d39d89452fdaa5\": rpc error: code = NotFound desc = could not find container \"702a622173a4630c5bd54e3e19145cf6f1601c7bef627fed08d39d89452fdaa5\": container with ID starting with 702a622173a4630c5bd54e3e19145cf6f1601c7bef627fed08d39d89452fdaa5 not found: ID does not exist"
Oct 31 00:53:10.987082 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.987052    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"]
Oct 31 00:53:10.991094 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:10.991065    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-cjpms"]
Oct 31 00:53:11.923223 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:11.923183    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ff44805-0009-41dc-9415-cc05817837ea" path="/var/lib/kubelet/pods/5ff44805-0009-41dc-9415-cc05817837ea/volumes"
Oct 31 00:53:12.785841 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:12.785800    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9"]
Oct 31 00:53:12.786072 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:12.786043    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9" podUID="58153479-6f09-44cf-bdd2-c7eb57580896" containerName="maas-api" containerID="cri-o://39aaf07926076e15a7c1513db1ab31bddace37e9408cd7cd171a78c96d681797" gracePeriod=30
Oct 31 00:53:13.273702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:13.273304    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9"
Oct 31 00:53:13.522805 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:13.520743    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58153479-6f09-44cf-bdd2-c7eb57580896-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58153479-6f09-44cf-bdd2-c7eb57580896-maas-api-tls\") pod \"58153479-6f09-44cf-bdd2-c7eb57580896\" (UID: \"58153479-6f09-44cf-bdd2-c7eb57580896\") "
Oct 31 00:53:13.522805 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:13.520946    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58153479-6f09-44cf-bdd2-c7eb57580896-metrics-certs\" (UniqueName: \"kubernetes.io/secret/58153479-6f09-44cf-bdd2-c7eb57580896-metrics-certs\") pod \"58153479-6f09-44cf-bdd2-c7eb57580896\" (UID: \"58153479-6f09-44cf-bdd2-c7eb57580896\") "
Oct 31 00:53:13.522805 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:13.521082    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58153479-6f09-44cf-bdd2-c7eb57580896-kube-api-access-ss4q4\" (UniqueName: \"kubernetes.io/projected/58153479-6f09-44cf-bdd2-c7eb57580896-kube-api-access-ss4q4\") pod \"58153479-6f09-44cf-bdd2-c7eb57580896\" (UID: \"58153479-6f09-44cf-bdd2-c7eb57580896\") "
Oct 31 00:53:13.522805 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:13.521145    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58153479-6f09-44cf-bdd2-c7eb57580896-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58153479-6f09-44cf-bdd2-c7eb57580896-openshift-service-ca\") pod \"58153479-6f09-44cf-bdd2-c7eb57580896\" (UID: \"58153479-6f09-44cf-bdd2-c7eb57580896\") "
Oct 31 00:53:13.522805 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:13.522181    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58153479-6f09-44cf-bdd2-c7eb57580896-openshift-service-ca" pod "58153479-6f09-44cf-bdd2-c7eb57580896" (UID: "58153479-6f09-44cf-bdd2-c7eb57580896"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:53:13.539938 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:13.539870    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58153479-6f09-44cf-bdd2-c7eb57580896-maas-api-tls" pod "58153479-6f09-44cf-bdd2-c7eb57580896" (UID: "58153479-6f09-44cf-bdd2-c7eb57580896"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:53:13.540803 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:13.540773    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58153479-6f09-44cf-bdd2-c7eb57580896-kube-api-access-ss4q4" pod "58153479-6f09-44cf-bdd2-c7eb57580896" (UID: "58153479-6f09-44cf-bdd2-c7eb57580896"). InnerVolumeSpecName "kube-api-access-ss4q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:53:13.548452 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:13.548410    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58153479-6f09-44cf-bdd2-c7eb57580896-metrics-certs" pod "58153479-6f09-44cf-bdd2-c7eb57580896" (UID: "58153479-6f09-44cf-bdd2-c7eb57580896"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:53:13.623603 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:13.623572    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58153479-6f09-44cf-bdd2-c7eb57580896-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:53:13.623603 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:13.623599    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58153479-6f09-44cf-bdd2-c7eb57580896-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:53:13.623603 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:13.623608    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss4q4\" (UniqueName: \"kubernetes.io/projected/58153479-6f09-44cf-bdd2-c7eb57580896-kube-api-access-ss4q4\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:53:13.623899 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:13.623620    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58153479-6f09-44cf-bdd2-c7eb57580896-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:53:13.898703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:13.898660    4327 generic.go:356] "Generic (PLEG): container finished" podID="58153479-6f09-44cf-bdd2-c7eb57580896" containerID="39aaf07926076e15a7c1513db1ab31bddace37e9408cd7cd171a78c96d681797" exitCode=0
Oct 31 00:53:13.898882 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:13.898737    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9" event={"ID":"58153479-6f09-44cf-bdd2-c7eb57580896","Type":"ContainerDied","Data":"39aaf07926076e15a7c1513db1ab31bddace37e9408cd7cd171a78c96d681797"}
Oct 31 00:53:13.898882 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:13.898799    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9" event={"ID":"58153479-6f09-44cf-bdd2-c7eb57580896","Type":"ContainerDied","Data":"2a2f96d439ffa74173fa1ec95ee8484cdec2fb75989406f43c19047f46a51ba2"}
Oct 31 00:53:13.898882 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:13.898820    4327 scope.go:122] "RemoveContainer" containerID="39aaf07926076e15a7c1513db1ab31bddace37e9408cd7cd171a78c96d681797"
Oct 31 00:53:13.898882 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:13.898746    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9"
Oct 31 00:53:13.910918 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:13.910892    4327 scope.go:122] "RemoveContainer" containerID="39aaf07926076e15a7c1513db1ab31bddace37e9408cd7cd171a78c96d681797"
Oct 31 00:53:13.911231 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:53:13.911207    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39aaf07926076e15a7c1513db1ab31bddace37e9408cd7cd171a78c96d681797\": container with ID starting with 39aaf07926076e15a7c1513db1ab31bddace37e9408cd7cd171a78c96d681797 not found: ID does not exist" containerID="39aaf07926076e15a7c1513db1ab31bddace37e9408cd7cd171a78c96d681797"
Oct 31 00:53:13.911306 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:13.911238    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39aaf07926076e15a7c1513db1ab31bddace37e9408cd7cd171a78c96d681797"} err="failed to get container status \"39aaf07926076e15a7c1513db1ab31bddace37e9408cd7cd171a78c96d681797\": rpc error: code = NotFound desc = could not find container \"39aaf07926076e15a7c1513db1ab31bddace37e9408cd7cd171a78c96d681797\": container with ID starting with 39aaf07926076e15a7c1513db1ab31bddace37e9408cd7cd171a78c96d681797 not found: ID does not exist"
Oct 31 00:53:13.979093 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:13.979063    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9"]
Oct 31 00:53:14.014257 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:14.014222    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d21c7-746f65c9cd-xw7k9"]
Oct 31 00:53:15.294894 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:53:15.294855    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:53:15.630157 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:15.630116    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf" podUID="99f7b791-5665-432d-bf5d-71a0553094fd" containerName="payload-processing" containerID="cri-o://949aaea68e12af5240f4a1b6f64c8467772cd4a0a885feb38bc9f9ae34530d90" gracePeriod=25
Oct 31 00:53:15.669541 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:15.669504    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54" podUID="e15553ed-f60a-42d7-9fea-2daa2a5c5360" containerName="payload-pre-processing" containerID="cri-o://8793a1bcfac933832ee75be879c99e3812c8564af5c117c78c0da91fdf5d03a2" gracePeriod=25
Oct 31 00:53:15.935653 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:15.935454    4327 generic.go:356] "Generic (PLEG): container finished" podID="e15553ed-f60a-42d7-9fea-2daa2a5c5360" containerID="8793a1bcfac933832ee75be879c99e3812c8564af5c117c78c0da91fdf5d03a2" exitCode=0
Oct 31 00:53:15.937841 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:15.937087    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58153479-6f09-44cf-bdd2-c7eb57580896" path="/var/lib/kubelet/pods/58153479-6f09-44cf-bdd2-c7eb57580896/volumes"
Oct 31 00:53:15.937841 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:15.937583    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54" event={"ID":"e15553ed-f60a-42d7-9fea-2daa2a5c5360","Type":"ContainerDied","Data":"8793a1bcfac933832ee75be879c99e3812c8564af5c117c78c0da91fdf5d03a2"}
Oct 31 00:53:15.938452 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:15.938431    4327 generic.go:356] "Generic (PLEG): container finished" podID="99f7b791-5665-432d-bf5d-71a0553094fd" containerID="949aaea68e12af5240f4a1b6f64c8467772cd4a0a885feb38bc9f9ae34530d90" exitCode=0
Oct 31 00:53:15.938634 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:15.938494    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf" event={"ID":"99f7b791-5665-432d-bf5d-71a0553094fd","Type":"ContainerDied","Data":"949aaea68e12af5240f4a1b6f64c8467772cd4a0a885feb38bc9f9ae34530d90"}
Oct 31 00:53:15.964423 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:15.964393    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf"
Oct 31 00:53:16.003646 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:16.003620    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54"
Oct 31 00:53:16.069721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:16.067106    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99f7b791-5665-432d-bf5d-71a0553094fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99f7b791-5665-432d-bf5d-71a0553094fd-plugins-config-volume\") pod \"99f7b791-5665-432d-bf5d-71a0553094fd\" (UID: \"99f7b791-5665-432d-bf5d-71a0553094fd\") "
Oct 31 00:53:16.069721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:16.067184    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99f7b791-5665-432d-bf5d-71a0553094fd-kube-api-access-mdrgl\" (UniqueName: \"kubernetes.io/projected/99f7b791-5665-432d-bf5d-71a0553094fd-kube-api-access-mdrgl\") pod \"99f7b791-5665-432d-bf5d-71a0553094fd\" (UID: \"99f7b791-5665-432d-bf5d-71a0553094fd\") "
Oct 31 00:53:16.069721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:16.069296    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99f7b791-5665-432d-bf5d-71a0553094fd-plugins-config-volume" pod "99f7b791-5665-432d-bf5d-71a0553094fd" (UID: "99f7b791-5665-432d-bf5d-71a0553094fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:53:16.069721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:16.069661    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99f7b791-5665-432d-bf5d-71a0553094fd-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:53:16.077720 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:16.074181    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f7b791-5665-432d-bf5d-71a0553094fd-kube-api-access-mdrgl" pod "99f7b791-5665-432d-bf5d-71a0553094fd" (UID: "99f7b791-5665-432d-bf5d-71a0553094fd"). InnerVolumeSpecName "kube-api-access-mdrgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:53:16.170982 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:16.170895    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e15553ed-f60a-42d7-9fea-2daa2a5c5360-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15553ed-f60a-42d7-9fea-2daa2a5c5360-plugins-config-volume\") pod \"e15553ed-f60a-42d7-9fea-2daa2a5c5360\" (UID: \"e15553ed-f60a-42d7-9fea-2daa2a5c5360\") "
Oct 31 00:53:16.171232 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:16.171214    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e15553ed-f60a-42d7-9fea-2daa2a5c5360-kube-api-access-66tbz\" (UniqueName: \"kubernetes.io/projected/e15553ed-f60a-42d7-9fea-2daa2a5c5360-kube-api-access-66tbz\") pod \"e15553ed-f60a-42d7-9fea-2daa2a5c5360\" (UID: \"e15553ed-f60a-42d7-9fea-2daa2a5c5360\") "
Oct 31 00:53:16.171283 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:16.171236    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e15553ed-f60a-42d7-9fea-2daa2a5c5360-plugins-config-volume" pod "e15553ed-f60a-42d7-9fea-2daa2a5c5360" (UID: "e15553ed-f60a-42d7-9fea-2daa2a5c5360"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:53:16.171898 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:16.171874    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdrgl\" (UniqueName: \"kubernetes.io/projected/99f7b791-5665-432d-bf5d-71a0553094fd-kube-api-access-mdrgl\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:53:16.171898 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:16.171896    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15553ed-f60a-42d7-9fea-2daa2a5c5360-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:53:16.173427 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:16.173400    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e15553ed-f60a-42d7-9fea-2daa2a5c5360-kube-api-access-66tbz" pod "e15553ed-f60a-42d7-9fea-2daa2a5c5360" (UID: "e15553ed-f60a-42d7-9fea-2daa2a5c5360"). InnerVolumeSpecName "kube-api-access-66tbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:53:16.273315 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:16.273278    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66tbz\" (UniqueName: \"kubernetes.io/projected/e15553ed-f60a-42d7-9fea-2daa2a5c5360-kube-api-access-66tbz\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:53:16.950031 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:16.949954    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54"
Oct 31 00:53:16.950488 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:16.949949    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54" event={"ID":"e15553ed-f60a-42d7-9fea-2daa2a5c5360","Type":"ContainerDied","Data":"4857753e8e4f3d8a67bca1c6397d017cd8dc56afe79a9fbcd58e9bc683094c31"}
Oct 31 00:53:16.950488 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:16.950108    4327 scope.go:122] "RemoveContainer" containerID="8793a1bcfac933832ee75be879c99e3812c8564af5c117c78c0da91fdf5d03a2"
Oct 31 00:53:16.951894 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:16.951862    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf" event={"ID":"99f7b791-5665-432d-bf5d-71a0553094fd","Type":"ContainerDied","Data":"7dd15d41b2a4586e0f06be770ac9d53ba03f461de7920308f8d0e36dcc05ae36"}
Oct 31 00:53:16.951894 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:16.951884    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf"
Oct 31 00:53:16.962989 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:16.962970    4327 scope.go:122] "RemoveContainer" containerID="949aaea68e12af5240f4a1b6f64c8467772cd4a0a885feb38bc9f9ae34530d90"
Oct 31 00:53:17.012769 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.012735    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf"]
Oct 31 00:53:17.072710 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.066962    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-68f412-5c85fb46cc-nhxzf"]
Oct 31 00:53:17.140763 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.140729    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54"]
Oct 31 00:53:17.158720 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.158000    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-68f412-9bf56d555-dcm54"]
Oct 31 00:53:17.404035 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.403999    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"]
Oct 31 00:53:17.404872 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.404856    4327 cpu_manager.go:414] "containerMap: removing container" podUID="58153479-6f09-44cf-bdd2-c7eb57580896" containerName="maas-api"
Oct 31 00:53:17.404928 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.404874    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58153479-6f09-44cf-bdd2-c7eb57580896" containerName="maas-api"
Oct 31 00:53:17.404928 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.404896    4327 cpu_manager.go:414] "containerMap: removing container" podUID="99f7b791-5665-432d-bf5d-71a0553094fd" containerName="payload-processing"
Oct 31 00:53:17.404928 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.404901    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99f7b791-5665-432d-bf5d-71a0553094fd" containerName="payload-processing"
Oct 31 00:53:17.405046 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.404944    4327 cpu_manager.go:414] "containerMap: removing container" podUID="5ff44805-0009-41dc-9415-cc05817837ea" containerName="istio-proxy"
Oct 31 00:53:17.405046 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.404950    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff44805-0009-41dc-9415-cc05817837ea" containerName="istio-proxy"
Oct 31 00:53:17.405046 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.404972    4327 cpu_manager.go:414] "containerMap: removing container" podUID="e15553ed-f60a-42d7-9fea-2daa2a5c5360" containerName="payload-pre-processing"
Oct 31 00:53:17.405046 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.404977    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e15553ed-f60a-42d7-9fea-2daa2a5c5360" containerName="payload-pre-processing"
Oct 31 00:53:17.405168 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.405100    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="99f7b791-5665-432d-bf5d-71a0553094fd" containerName="payload-processing"
Oct 31 00:53:17.405168 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.405115    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff44805-0009-41dc-9415-cc05817837ea" containerName="istio-proxy"
Oct 31 00:53:17.405168 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.405123    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="e15553ed-f60a-42d7-9fea-2daa2a5c5360" containerName="payload-pre-processing"
Oct 31 00:53:17.405168 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.405144    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="58153479-6f09-44cf-bdd2-c7eb57580896" containerName="maas-api"
Oct 31 00:53:17.410416 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.410395    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.422473 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.422443    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-workload-certs\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.422641 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.422485    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-envoy\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.422641 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.422509    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-token\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.422641 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.422532    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-credential-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.422641 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.422551    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9tmj\" (UniqueName: \"kubernetes.io/projected/4511417e-7a7f-4671-969b-b87f7668c3f4-kube-api-access-d9tmj\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.422641 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.422578    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-workload-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.422641 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.422594    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4511417e-7a7f-4671-969b-b87f7668c3f4-istiod-ca-cert\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.422641 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.422632    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-podinfo\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.422641 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.422647    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-data\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.436521 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.436495    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"]
Oct 31 00:53:17.439361 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.439338    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-82d887-openshift-default-dockercfg-zdn5c\""
Oct 31 00:53:17.523855 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.523822    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-podinfo\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.523855 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.523856    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-data\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.524113 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.524093    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-workload-certs\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.524179 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.524133    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-envoy\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.524278 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.524255    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-token\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.524336 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.524305    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-credential-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.524392 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.524332    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-data\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.524392 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.524338    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9tmj\" (UniqueName: \"kubernetes.io/projected/4511417e-7a7f-4671-969b-b87f7668c3f4-kube-api-access-d9tmj\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.524498 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.524421    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-workload-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.524498 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.524454    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4511417e-7a7f-4671-969b-b87f7668c3f4-istiod-ca-cert\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.524626 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.524606    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-credential-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.524749 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.524670    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-workload-certs\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.524813 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.524760    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-workload-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.525170 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.525147    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4511417e-7a7f-4671-969b-b87f7668c3f4-istiod-ca-cert\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.526895 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.526872    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-envoy\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.527068 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.527045    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-podinfo\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.547689 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.547644    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-token\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.547870 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.547830    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9tmj\" (UniqueName: \"kubernetes.io/projected/4511417e-7a7f-4671-969b-b87f7668c3f4-kube-api-access-d9tmj\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.729576 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.729489    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:17.922941 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.922909    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99f7b791-5665-432d-bf5d-71a0553094fd" path="/var/lib/kubelet/pods/99f7b791-5665-432d-bf5d-71a0553094fd/volumes"
Oct 31 00:53:17.923430 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:17.923414    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e15553ed-f60a-42d7-9fea-2daa2a5c5360" path="/var/lib/kubelet/pods/e15553ed-f60a-42d7-9fea-2daa2a5c5360/volumes"
Oct 31 00:53:18.086889 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:53:18.086850    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4511417e_7a7f_4671_969b_b87f7668c3f4.slice/crio-3fcb17c7d274cb96bb0ad96f6120943cfd01c77005e1945c18db73f32e8e7e77 WatchSource:0}: Error finding container 3fcb17c7d274cb96bb0ad96f6120943cfd01c77005e1945c18db73f32e8e7e77: Status 404 returned error can't find the container with id 3fcb17c7d274cb96bb0ad96f6120943cfd01c77005e1945c18db73f32e8e7e77
Oct 31 00:53:18.087602 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.087581    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"]
Oct 31 00:53:18.090725 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.090667    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:53:18.090824 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.090776    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:53:18.090874 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.090822    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:53:18.311998 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.311889    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"]
Oct 31 00:53:18.316837 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.316813    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.339423 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.339348    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-envoy\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.339423 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.339407    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k2c9\" (UniqueName: \"kubernetes.io/projected/242be892-3b92-44cb-a8ef-17f8ac581b4b-kube-api-access-6k2c9\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.339632 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.339455    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-workload-socket\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.339632 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.339494    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-token\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.339632 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.339538    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-credential-socket\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.339828 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.339755    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242be892-3b92-44cb-a8ef-17f8ac581b4b-istiod-ca-cert\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.339879 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.339834    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-data\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.339879 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.339862    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-workload-certs\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.339985 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.339890    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-podinfo\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.343612 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.343583    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b87c62-openshift-default-dockercfg-7wttk\""
Oct 31 00:53:18.359517 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.359478    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"]
Oct 31 00:53:18.454011 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.453960    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242be892-3b92-44cb-a8ef-17f8ac581b4b-istiod-ca-cert\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.454358 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.454329    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-data\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.454548 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.454528    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-workload-certs\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.454765 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.454745    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-podinfo\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.454991 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.454973    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-envoy\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.455128 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.455113    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k2c9\" (UniqueName: \"kubernetes.io/projected/242be892-3b92-44cb-a8ef-17f8ac581b4b-kube-api-access-6k2c9\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.455263 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.455245    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-workload-socket\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.455385 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.455370    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-token\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.455517 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.455502    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-credential-socket\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.470941 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.469131    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-credential-socket\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.471171 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.469787    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242be892-3b92-44cb-a8ef-17f8ac581b4b-istiod-ca-cert\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.471282 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.470032    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-data\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.471375 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.470243    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-workload-certs\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.481652 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.481618    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-workload-socket\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.574420 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.574338    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-envoy\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.613702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.608870    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-token\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.632711 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.627659    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-podinfo\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.665709 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.657458    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k2c9\" (UniqueName: \"kubernetes.io/projected/242be892-3b92-44cb-a8ef-17f8ac581b4b-kube-api-access-6k2c9\") pod \"e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") " pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:18.942063 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:18.941971    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:19.010035 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:19.009988    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp" event={"ID":"4511417e-7a7f-4671-969b-b87f7668c3f4","Type":"ContainerStarted","Data":"67323dbef922683532bd04de5e623c15f47f9867c5a31a275dc588e6c610cf42"}
Oct 31 00:53:19.010280 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:19.010258    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp" event={"ID":"4511417e-7a7f-4671-969b-b87f7668c3f4","Type":"ContainerStarted","Data":"3fcb17c7d274cb96bb0ad96f6120943cfd01c77005e1945c18db73f32e8e7e77"}
Oct 31 00:53:19.070886 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:19.070760    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp" podStartSLOduration=2.070743541 podStartE2EDuration="2.070743541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:19.061530301 +0000 UTC m=+2059.283565847" watchObservedRunningTime="2026-10-31 00:53:19.070743541 +0000 UTC m=+2059.292779144"
Oct 31 00:53:19.348540 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:19.348509    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"]
Oct 31 00:53:19.349755 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:53:19.349727    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod242be892_3b92_44cb_a8ef_17f8ac581b4b.slice/crio-cc89b27ca47b1bd5370be0d499bee2069e29f7dfe452e6bf5241973d3bec8782 WatchSource:0}: Error finding container cc89b27ca47b1bd5370be0d499bee2069e29f7dfe452e6bf5241973d3bec8782: Status 404 returned error can't find the container with id cc89b27ca47b1bd5370be0d499bee2069e29f7dfe452e6bf5241973d3bec8782
Oct 31 00:53:19.352875 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:19.352840    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:53:19.352990 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:19.352913    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:53:19.352990 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:19.352943    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:53:19.732937 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:19.732796    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:19.734579 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:19.734235    4327 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 31 00:53:19.734579 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:19.734288    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp" podUID="4511417e-7a7f-4671-969b-b87f7668c3f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 31 00:53:20.059531 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:20.059435    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk" event={"ID":"242be892-3b92-44cb-a8ef-17f8ac581b4b","Type":"ContainerStarted","Data":"352a138edc15db7342152a37d38e4c6ff57b41b971fe361b2276fa3ee8960229"}
Oct 31 00:53:20.059531 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:20.059500    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk" event={"ID":"242be892-3b92-44cb-a8ef-17f8ac581b4b","Type":"ContainerStarted","Data":"cc89b27ca47b1bd5370be0d499bee2069e29f7dfe452e6bf5241973d3bec8782"}
Oct 31 00:53:20.107465 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:20.107413    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk" podStartSLOduration=2.107394854 podStartE2EDuration="2.107394854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:53:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:20.105921532 +0000 UTC m=+2060.327957094" watchObservedRunningTime="2026-10-31 00:53:20.107394854 +0000 UTC m=+2060.329430399"
Oct 31 00:53:20.741227 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:20.741183    4327 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 31 00:53:20.742271 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:20.741828    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp" podUID="4511417e-7a7f-4671-969b-b87f7668c3f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 31 00:53:20.942891 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:20.942846    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:20.949208 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:20.949182    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:21.070636 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:21.070503    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:21.071455 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:21.071433    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:53:21.733704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:21.731072    4327 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 31 00:53:21.733704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:21.731131    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp" podUID="4511417e-7a7f-4671-969b-b87f7668c3f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 31 00:53:22.736705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:22.736122    4327 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 31 00:53:22.736705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:22.736201    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp" podUID="4511417e-7a7f-4671-969b-b87f7668c3f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 31 00:53:23.755500 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:23.755454    4327 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 31 00:53:23.756145 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:23.755520    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp" podUID="4511417e-7a7f-4671-969b-b87f7668c3f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 31 00:53:24.741706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:24.739710    4327 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 31 00:53:24.741706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:24.739793    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp" podUID="4511417e-7a7f-4671-969b-b87f7668c3f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 31 00:53:25.735434 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:25.735405    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:25.735957 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:25.735937    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:25.736669 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:25.736647    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:53:26.503865 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:26.503834    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b"]
Oct 31 00:53:26.508626 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:26.508605    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b"
Oct 31 00:53:26.525431 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:26.525405    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b87c62-dockercfg-59p7n\""
Oct 31 00:53:26.529894 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:26.529870    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b"]
Oct 31 00:53:26.642703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:26.624442    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b87c62\""
Oct 31 00:53:26.662706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:26.650422    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b271c9fe-0a7d-4b81-b145-4db3f776e136-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b\" (UID: \"b271c9fe-0a7d-4b81-b145-4db3f776e136\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b"
Oct 31 00:53:26.727700 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:26.650585    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmnhr\" (UniqueName: \"kubernetes.io/projected/b271c9fe-0a7d-4b81-b145-4db3f776e136-kube-api-access-dmnhr\") pod \"payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b\" (UID: \"b271c9fe-0a7d-4b81-b145-4db3f776e136\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b"
Oct 31 00:53:26.777815 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:26.777724    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244"]
Oct 31 00:53:26.845174 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:26.841938    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b271c9fe-0a7d-4b81-b145-4db3f776e136-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b\" (UID: \"b271c9fe-0a7d-4b81-b145-4db3f776e136\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b"
Oct 31 00:53:26.845174 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:26.842088    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmnhr\" (UniqueName: \"kubernetes.io/projected/b271c9fe-0a7d-4b81-b145-4db3f776e136-kube-api-access-dmnhr\") pod \"payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b\" (UID: \"b271c9fe-0a7d-4b81-b145-4db3f776e136\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b"
Oct 31 00:53:26.845174 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:26.843643    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b271c9fe-0a7d-4b81-b145-4db3f776e136-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b\" (UID: \"b271c9fe-0a7d-4b81-b145-4db3f776e136\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b"
Oct 31 00:53:26.909759 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:26.908255    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244"]
Oct 31 00:53:26.910174 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:26.908593    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244"
Oct 31 00:53:26.928620 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:26.928591    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmnhr\" (UniqueName: \"kubernetes.io/projected/b271c9fe-0a7d-4b81-b145-4db3f776e136-kube-api-access-dmnhr\") pod \"payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b\" (UID: \"b271c9fe-0a7d-4b81-b145-4db3f776e136\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b"
Oct 31 00:53:27.045075 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:27.044974    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqqxk\" (UniqueName: \"kubernetes.io/projected/08dd4688-b889-43ab-bba9-58afcf509595-kube-api-access-gqqxk\") pod \"payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244\" (UID: \"08dd4688-b889-43ab-bba9-58afcf509595\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244"
Oct 31 00:53:27.045262 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:27.045217    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08dd4688-b889-43ab-bba9-58afcf509595-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244\" (UID: \"08dd4688-b889-43ab-bba9-58afcf509595\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244"
Oct 31 00:53:27.132409 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:27.132362    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b"
Oct 31 00:53:27.146571 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:27.146544    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqqxk\" (UniqueName: \"kubernetes.io/projected/08dd4688-b889-43ab-bba9-58afcf509595-kube-api-access-gqqxk\") pod \"payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244\" (UID: \"08dd4688-b889-43ab-bba9-58afcf509595\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244"
Oct 31 00:53:27.146731 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:27.146654    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08dd4688-b889-43ab-bba9-58afcf509595-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244\" (UID: \"08dd4688-b889-43ab-bba9-58afcf509595\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244"
Oct 31 00:53:27.147527 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:27.147508    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08dd4688-b889-43ab-bba9-58afcf509595-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244\" (UID: \"08dd4688-b889-43ab-bba9-58afcf509595\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244"
Oct 31 00:53:27.169999 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:27.169971    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqqxk\" (UniqueName: \"kubernetes.io/projected/08dd4688-b889-43ab-bba9-58afcf509595-kube-api-access-gqqxk\") pod \"payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244\" (UID: \"08dd4688-b889-43ab-bba9-58afcf509595\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244"
Oct 31 00:53:27.233241 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:27.232964    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244"
Oct 31 00:53:27.447231 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:27.447206    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b"]
Oct 31 00:53:27.448569 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:53:27.448542    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb271c9fe_0a7d_4b81_b145_4db3f776e136.slice/crio-fae356fb466cbfbdc79212eace2cbbce0abb5a24687a103fc20d9ec1a0ee4677 WatchSource:0}: Error finding container fae356fb466cbfbdc79212eace2cbbce0abb5a24687a103fc20d9ec1a0ee4677: Status 404 returned error can't find the container with id fae356fb466cbfbdc79212eace2cbbce0abb5a24687a103fc20d9ec1a0ee4677
Oct 31 00:53:27.562030 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:27.561991    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244"]
Oct 31 00:53:28.246995 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:28.246952    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b" event={"ID":"b271c9fe-0a7d-4b81-b145-4db3f776e136","Type":"ContainerStarted","Data":"5ff213c3a14eb61a4d1d7c884b58a7d8d7c60467c6f07a653baaa3eb9dec0aca"}
Oct 31 00:53:28.247523 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:28.247026    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b" event={"ID":"b271c9fe-0a7d-4b81-b145-4db3f776e136","Type":"ContainerStarted","Data":"fae356fb466cbfbdc79212eace2cbbce0abb5a24687a103fc20d9ec1a0ee4677"}
Oct 31 00:53:28.247523 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:28.247277    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b"
Oct 31 00:53:28.249578 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:28.249548    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244" event={"ID":"08dd4688-b889-43ab-bba9-58afcf509595","Type":"ContainerStarted","Data":"e38b5fee287278953f936f2ce53f4be4692b283798d218b66ec6de757611e4ab"}
Oct 31 00:53:28.249749 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:28.249600    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244" event={"ID":"08dd4688-b889-43ab-bba9-58afcf509595","Type":"ContainerStarted","Data":"b5fb7927f9dda5acc4143c188e928014d7af2437e177ce0e67b4870225e4833e"}
Oct 31 00:53:28.249749 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:28.249644    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244"
Oct 31 00:53:28.298882 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:28.298827    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b" podStartSLOduration=2.29881173 podStartE2EDuration="2.29881173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:28.296386928 +0000 UTC m=+2068.518422471" watchObservedRunningTime="2026-10-31 00:53:28.29881173 +0000 UTC m=+2068.520847272"
Oct 31 00:53:28.350555 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:28.350498    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244" podStartSLOduration=2.350484438 podStartE2EDuration="2.350484438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:28.348014759 +0000 UTC m=+2068.570050303" watchObservedRunningTime="2026-10-31 00:53:28.350484438 +0000 UTC m=+2068.572519980"
Oct 31 00:53:39.261452 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:39.261413    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244"
Oct 31 00:53:39.262057 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:39.261502    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b"
Oct 31 00:53:45.295904 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:53:45.295862    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:53:54.357606 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:54.356798    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:53:54.460730 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:53:54.452385    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:54:02.531288 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.531249    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"]
Oct 31 00:54:02.537737 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.537709    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.573416 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.573380    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"]
Oct 31 00:54:02.623473 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.623430    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-workload-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.623733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.623482    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb121139-61a4-498c-9ace-56d1394a8205-istiod-ca-cert\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.623733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.623527    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-credential-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.623733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.623560    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-istio-envoy\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.623733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.623588    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb121139-61a4-498c-9ace-56d1394a8205-istio-podinfo\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.623733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.623613    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-istio-data\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.623733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.623658    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wqqc\" (UniqueName: \"kubernetes.io/projected/cb121139-61a4-498c-9ace-56d1394a8205-kube-api-access-8wqqc\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.624061 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.623732    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb121139-61a4-498c-9ace-56d1394a8205-istio-token\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.624061 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.623857    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-workload-certs\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.725597 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.725558    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-workload-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.725792 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.725608    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb121139-61a4-498c-9ace-56d1394a8205-istiod-ca-cert\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.725792 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.725654    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-credential-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.725792 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.725733    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-istio-envoy\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.725792 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.725773    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb121139-61a4-498c-9ace-56d1394a8205-istio-podinfo\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.726011 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.725799    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-istio-data\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.726011 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.725873    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wqqc\" (UniqueName: \"kubernetes.io/projected/cb121139-61a4-498c-9ace-56d1394a8205-kube-api-access-8wqqc\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.726011 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.725904    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb121139-61a4-498c-9ace-56d1394a8205-istio-token\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.726165 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.726044    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-workload-certs\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.726225 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.726168    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-credential-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.726399 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.726368    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-workload-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.726519 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.726496    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-workload-certs\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.726616 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.726595    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb121139-61a4-498c-9ace-56d1394a8205-istiod-ca-cert\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.726695 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.726598    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-istio-data\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.728110 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.728090    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-istio-envoy\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.728474 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.728455    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb121139-61a4-498c-9ace-56d1394a8205-istio-podinfo\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.751438 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.751405    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb121139-61a4-498c-9ace-56d1394a8205-istio-token\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.751566 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.751468    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wqqc\" (UniqueName: \"kubernetes.io/projected/cb121139-61a4-498c-9ace-56d1394a8205-kube-api-access-8wqqc\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:02.858085 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:02.858049    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:03.204543 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:54:03.204510    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb121139_61a4_498c_9ace_56d1394a8205.slice/crio-dbe9564d0713889cbdb389f80979aec6fe54e201826a1e32a05e875cfb2c5da8 WatchSource:0}: Error finding container dbe9564d0713889cbdb389f80979aec6fe54e201826a1e32a05e875cfb2c5da8: Status 404 returned error can't find the container with id dbe9564d0713889cbdb389f80979aec6fe54e201826a1e32a05e875cfb2c5da8
Oct 31 00:54:03.204996 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:03.204977    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"]
Oct 31 00:54:03.207572 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:03.207537    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:54:03.207712 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:03.207606    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:54:03.207712 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:03.207635    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:54:03.854706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:03.854508    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97" event={"ID":"cb121139-61a4-498c-9ace-56d1394a8205","Type":"ContainerStarted","Data":"410eb56ece4aeb791fb8c447b972e43e90ab870f7329a9880661470a56e0d1df"}
Oct 31 00:54:03.854706 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:03.854580    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97" event={"ID":"cb121139-61a4-498c-9ace-56d1394a8205","Type":"ContainerStarted","Data":"dbe9564d0713889cbdb389f80979aec6fe54e201826a1e32a05e875cfb2c5da8"}
Oct 31 00:54:03.965871 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:03.965801    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97" podStartSLOduration=1.965775839 podStartE2EDuration="1.965775839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:03.933256111 +0000 UTC m=+2104.155291654" watchObservedRunningTime="2026-10-31 00:54:03.965775839 +0000 UTC m=+2104.187811386"
Oct 31 00:54:04.858421 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:04.858388    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:04.860392 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:04.860361    4327 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 31 00:54:04.860539 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:04.860423    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97" podUID="cb121139-61a4-498c-9ace-56d1394a8205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 31 00:54:05.867112 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:05.865109    4327 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 31 00:54:05.867112 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:05.865208    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97" podUID="cb121139-61a4-498c-9ace-56d1394a8205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 31 00:54:06.087405 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:06.068186    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr"]
Oct 31 00:54:06.087405 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:06.068438    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr" podUID="7cd44166-6fbc-4252-919d-42503da5bb55" containerName="maas-api" containerID="cri-o://29d4a0b704508f53f3fc1ae36b21d1147c95178a7736787614faf8e85556369a" gracePeriod=30
Oct 31 00:54:06.860414 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:06.860374    4327 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 31 00:54:06.860745 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:06.860714    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97" podUID="cb121139-61a4-498c-9ace-56d1394a8205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 31 00:54:07.012719 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:07.012344    4327 generic.go:356] "Generic (PLEG): container finished" podID="7cd44166-6fbc-4252-919d-42503da5bb55" containerID="29d4a0b704508f53f3fc1ae36b21d1147c95178a7736787614faf8e85556369a" exitCode=0
Oct 31 00:54:07.012719 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:07.012465    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr" event={"ID":"7cd44166-6fbc-4252-919d-42503da5bb55","Type":"ContainerDied","Data":"29d4a0b704508f53f3fc1ae36b21d1147c95178a7736787614faf8e85556369a"}
Oct 31 00:54:07.585922 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:07.585861    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr"
Oct 31 00:54:07.685154 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:07.685111    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cd44166-6fbc-4252-919d-42503da5bb55-kube-api-access-ldx46\" (UniqueName: \"kubernetes.io/projected/7cd44166-6fbc-4252-919d-42503da5bb55-kube-api-access-ldx46\") pod \"7cd44166-6fbc-4252-919d-42503da5bb55\" (UID: \"7cd44166-6fbc-4252-919d-42503da5bb55\") "
Oct 31 00:54:07.685357 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:07.685313    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cd44166-6fbc-4252-919d-42503da5bb55-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cd44166-6fbc-4252-919d-42503da5bb55-metrics-certs\") pod \"7cd44166-6fbc-4252-919d-42503da5bb55\" (UID: \"7cd44166-6fbc-4252-919d-42503da5bb55\") "
Oct 31 00:54:07.685533 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:07.685515    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cd44166-6fbc-4252-919d-42503da5bb55-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cd44166-6fbc-4252-919d-42503da5bb55-maas-api-tls\") pod \"7cd44166-6fbc-4252-919d-42503da5bb55\" (UID: \"7cd44166-6fbc-4252-919d-42503da5bb55\") "
Oct 31 00:54:07.685618 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:07.685602    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cd44166-6fbc-4252-919d-42503da5bb55-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cd44166-6fbc-4252-919d-42503da5bb55-openshift-service-ca\") pod \"7cd44166-6fbc-4252-919d-42503da5bb55\" (UID: \"7cd44166-6fbc-4252-919d-42503da5bb55\") "
Oct 31 00:54:07.687243 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:07.687213    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cd44166-6fbc-4252-919d-42503da5bb55-openshift-service-ca" pod "7cd44166-6fbc-4252-919d-42503da5bb55" (UID: "7cd44166-6fbc-4252-919d-42503da5bb55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:54:07.688007 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:07.687976    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cd44166-6fbc-4252-919d-42503da5bb55-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:07.689511 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:07.689485    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cd44166-6fbc-4252-919d-42503da5bb55-maas-api-tls" pod "7cd44166-6fbc-4252-919d-42503da5bb55" (UID: "7cd44166-6fbc-4252-919d-42503da5bb55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:54:07.689620 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:07.689575    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cd44166-6fbc-4252-919d-42503da5bb55-metrics-certs" pod "7cd44166-6fbc-4252-919d-42503da5bb55" (UID: "7cd44166-6fbc-4252-919d-42503da5bb55"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:54:07.689698 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:07.689615    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd44166-6fbc-4252-919d-42503da5bb55-kube-api-access-ldx46" pod "7cd44166-6fbc-4252-919d-42503da5bb55" (UID: "7cd44166-6fbc-4252-919d-42503da5bb55"). InnerVolumeSpecName "kube-api-access-ldx46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:54:07.789133 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:07.789094    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cd44166-6fbc-4252-919d-42503da5bb55-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:07.789133 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:07.789127    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldx46\" (UniqueName: \"kubernetes.io/projected/7cd44166-6fbc-4252-919d-42503da5bb55-kube-api-access-ldx46\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:07.789133 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:07.789137    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cd44166-6fbc-4252-919d-42503da5bb55-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:07.862557 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:07.862101    4327 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 31 00:54:07.862557 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:07.862163    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97" podUID="cb121139-61a4-498c-9ace-56d1394a8205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 31 00:54:08.108020 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:08.107989    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr"
Oct 31 00:54:08.182943 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:08.182651    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr" event={"ID":"7cd44166-6fbc-4252-919d-42503da5bb55","Type":"ContainerDied","Data":"6213cddf981a29678305db032a2654a7ef6b54b03f1a570c6fe167a117a28527"}
Oct 31 00:54:08.183136 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:08.182934    4327 scope.go:122] "RemoveContainer" containerID="29d4a0b704508f53f3fc1ae36b21d1147c95178a7736787614faf8e85556369a"
Oct 31 00:54:08.265913 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:08.265873    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr"]
Oct 31 00:54:08.272153 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:08.272118    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-68f412-75cd6cdcc8-887tr"]
Oct 31 00:54:08.863955 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:08.863904    4327 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 31 00:54:08.864271 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:08.864243    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97" podUID="cb121139-61a4-498c-9ace-56d1394a8205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 31 00:54:09.867041 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:09.867006    4327 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 31 00:54:09.867642 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:09.867605    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97" podUID="cb121139-61a4-498c-9ace-56d1394a8205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 31 00:54:09.991771 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:09.991723    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cd44166-6fbc-4252-919d-42503da5bb55" path="/var/lib/kubelet/pods/7cd44166-6fbc-4252-919d-42503da5bb55/volumes"
Oct 31 00:54:10.860477 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:10.859910    4327 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 31 00:54:10.860477 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:10.860018    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97" podUID="cb121139-61a4-498c-9ace-56d1394a8205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 31 00:54:11.988120 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:11.986767    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:11.990509 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:11.988595    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:11.997530 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:11.997012    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:54:12.093835 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.093800    4327 patch_prober.go:29] interesting pod/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:54:12.094021 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.093869    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d" podUID="79f228f5-1947-426c-9e4d-4f300f43511e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:54:12.568944 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.568909    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"]
Oct 31 00:54:12.569887 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.569862    4327 cpu_manager.go:414] "containerMap: removing container" podUID="7cd44166-6fbc-4252-919d-42503da5bb55" containerName="maas-api"
Oct 31 00:54:12.569887 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.569886    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cd44166-6fbc-4252-919d-42503da5bb55" containerName="maas-api"
Oct 31 00:54:12.570103 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.570088    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cd44166-6fbc-4252-919d-42503da5bb55" containerName="maas-api"
Oct 31 00:54:12.574029 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.574009    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"
Oct 31 00:54:12.590647 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.590620    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4d5621\"/\"default-dockercfg-pdbhc\""
Oct 31 00:54:12.600999 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.600971    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4d5621\"/\"test-model-4d5621-kserve-self-signed-certs\""
Oct 31 00:54:12.604272 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.604240    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"]
Oct 31 00:54:12.640236 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.640201    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-model-cache\") pod \"test-model-4d5621-kserve-75549b8999-q24t6\" (UID: \"1b410769-3505-4588-9255-fcca23401c3e\") " pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"
Oct 31 00:54:12.640441 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.640344    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b410769-3505-4588-9255-fcca23401c3e-tls-certs\") pod \"test-model-4d5621-kserve-75549b8999-q24t6\" (UID: \"1b410769-3505-4588-9255-fcca23401c3e\") " pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"
Oct 31 00:54:12.640441 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.640417    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-home\") pod \"test-model-4d5621-kserve-75549b8999-q24t6\" (UID: \"1b410769-3505-4588-9255-fcca23401c3e\") " pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"
Oct 31 00:54:12.640555 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.640533    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-dshm\") pod \"test-model-4d5621-kserve-75549b8999-q24t6\" (UID: \"1b410769-3505-4588-9255-fcca23401c3e\") " pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"
Oct 31 00:54:12.640722 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.640696    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsctq\" (UniqueName: \"kubernetes.io/projected/1b410769-3505-4588-9255-fcca23401c3e-kube-api-access-nsctq\") pod \"test-model-4d5621-kserve-75549b8999-q24t6\" (UID: \"1b410769-3505-4588-9255-fcca23401c3e\") " pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"
Oct 31 00:54:12.640790 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.640774    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-tmp-dir\") pod \"test-model-4d5621-kserve-75549b8999-q24t6\" (UID: \"1b410769-3505-4588-9255-fcca23401c3e\") " pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"
Oct 31 00:54:12.680766 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.680732    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4d5621\"/\"kube-root-ca.crt\""
Oct 31 00:54:12.680934 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.680866    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4d5621\"/\"openshift-service-ca.crt\""
Oct 31 00:54:12.742382 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.742346    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b410769-3505-4588-9255-fcca23401c3e-tls-certs\") pod \"test-model-4d5621-kserve-75549b8999-q24t6\" (UID: \"1b410769-3505-4588-9255-fcca23401c3e\") " pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"
Oct 31 00:54:12.742555 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.742408    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-home\") pod \"test-model-4d5621-kserve-75549b8999-q24t6\" (UID: \"1b410769-3505-4588-9255-fcca23401c3e\") " pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"
Oct 31 00:54:12.742555 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.742466    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-dshm\") pod \"test-model-4d5621-kserve-75549b8999-q24t6\" (UID: \"1b410769-3505-4588-9255-fcca23401c3e\") " pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"
Oct 31 00:54:12.742650 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.742568    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsctq\" (UniqueName: \"kubernetes.io/projected/1b410769-3505-4588-9255-fcca23401c3e-kube-api-access-nsctq\") pod \"test-model-4d5621-kserve-75549b8999-q24t6\" (UID: \"1b410769-3505-4588-9255-fcca23401c3e\") " pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"
Oct 31 00:54:12.742650 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.742620    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-tmp-dir\") pod \"test-model-4d5621-kserve-75549b8999-q24t6\" (UID: \"1b410769-3505-4588-9255-fcca23401c3e\") " pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"
Oct 31 00:54:12.742835 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.742814    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-model-cache\") pod \"test-model-4d5621-kserve-75549b8999-q24t6\" (UID: \"1b410769-3505-4588-9255-fcca23401c3e\") " pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"
Oct 31 00:54:12.743089 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.743058    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-home\") pod \"test-model-4d5621-kserve-75549b8999-q24t6\" (UID: \"1b410769-3505-4588-9255-fcca23401c3e\") " pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"
Oct 31 00:54:12.743210 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.743058    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-tmp-dir\") pod \"test-model-4d5621-kserve-75549b8999-q24t6\" (UID: \"1b410769-3505-4588-9255-fcca23401c3e\") " pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"
Oct 31 00:54:12.743278 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.743205    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-model-cache\") pod \"test-model-4d5621-kserve-75549b8999-q24t6\" (UID: \"1b410769-3505-4588-9255-fcca23401c3e\") " pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"
Oct 31 00:54:12.745197 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.745167    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-dshm\") pod \"test-model-4d5621-kserve-75549b8999-q24t6\" (UID: \"1b410769-3505-4588-9255-fcca23401c3e\") " pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"
Oct 31 00:54:12.745377 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.745355    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b410769-3505-4588-9255-fcca23401c3e-tls-certs\") pod \"test-model-4d5621-kserve-75549b8999-q24t6\" (UID: \"1b410769-3505-4588-9255-fcca23401c3e\") " pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"
Oct 31 00:54:12.766815 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.766780    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsctq\" (UniqueName: \"kubernetes.io/projected/1b410769-3505-4588-9255-fcca23401c3e-kube-api-access-nsctq\") pod \"test-model-4d5621-kserve-75549b8999-q24t6\" (UID: \"1b410769-3505-4588-9255-fcca23401c3e\") " pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"
Oct 31 00:54:12.910701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:12.902086    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"
Oct 31 00:54:13.350874 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:54:13.350833    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b410769_3505_4588_9255_fcca23401c3e.slice/crio-2b7072b23be82fbed85aaf297c4906fcf88d62759365a49e5c0cc931e094cb46 WatchSource:0}: Error finding container 2b7072b23be82fbed85aaf297c4906fcf88d62759365a49e5c0cc931e094cb46: Status 404 returned error can't find the container with id 2b7072b23be82fbed85aaf297c4906fcf88d62759365a49e5c0cc931e094cb46
Oct 31 00:54:13.351238 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:13.350917    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"]
Oct 31 00:54:14.313396 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:14.313357    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6" event={"ID":"1b410769-3505-4588-9255-fcca23401c3e","Type":"ContainerStarted","Data":"4e38930e5031186c751a3ddcba7d08966fd73b83238618c70e2d6acd0e856404"}
Oct 31 00:54:14.313396 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:14.313407    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6" event={"ID":"1b410769-3505-4588-9255-fcca23401c3e","Type":"ContainerStarted","Data":"2b7072b23be82fbed85aaf297c4906fcf88d62759365a49e5c0cc931e094cb46"}
Oct 31 00:54:14.357815 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:14.357743    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6" podStartSLOduration=2.357705551 podStartE2EDuration="2.357705551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:14.355996397 +0000 UTC m=+2114.578031940" watchObservedRunningTime="2026-10-31 00:54:14.357705551 +0000 UTC m=+2114.579741092"
Oct 31 00:54:14.902842 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:14.902809    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"
Oct 31 00:54:14.917250 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:14.917215    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"]
Oct 31 00:54:14.917649 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:14.917611    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv" podUID="400fc179-6148-4af6-b069-88f843600416" containerName="istio-proxy" containerID="cri-o://71d469b3c678d63ecb92fe3295e938187b187d4a1be0b0ba562a64b90f03e5f8" gracePeriod=30
Oct 31 00:54:15.349703 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:54:15.349630    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:54:17.354177 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:17.354143    4327 generic.go:356] "Generic (PLEG): container finished" podID="79f228f5-1947-426c-9e4d-4f300f43511e" containerID="6a50b7d905e033190187c39065ee1df2811ab7c476e902c359e8fdedd4b5b8ea" exitCode=0
Oct 31 00:54:17.354639 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:17.354217    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d" event={"ID":"79f228f5-1947-426c-9e4d-4f300f43511e","Type":"ContainerDied","Data":"6a50b7d905e033190187c39065ee1df2811ab7c476e902c359e8fdedd4b5b8ea"}
Oct 31 00:54:17.354723 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:17.354713    4327 scope.go:122] "RemoveContainer" containerID="6a50b7d905e033190187c39065ee1df2811ab7c476e902c359e8fdedd4b5b8ea"
Oct 31 00:54:17.356598 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:17.356561    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:54:17.356752 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:17.356653    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:54:17.356828 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:17.356746    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:54:18.364274 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:18.364240    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d" event={"ID":"79f228f5-1947-426c-9e4d-4f300f43511e","Type":"ContainerStarted","Data":"1c7268daabd20de5cb1fc2b74dab2150c455e7a3a1d83323d5c65bc2b519388e"}
Oct 31 00:54:18.680887 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:18.680799    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:54:18.681067 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:18.681010    4327 patch_prober.go:29] interesting pod/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 00:54:18.681137 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:18.681061    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d" podUID="79f228f5-1947-426c-9e4d-4f300f43511e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 00:54:19.680393 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:19.680359    4327 patch_prober.go:29] interesting pod/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 00:54:19.680929 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:19.680447    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d" podUID="79f228f5-1947-426c-9e4d-4f300f43511e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 00:54:20.384479 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.384446    4327 generic.go:356] "Generic (PLEG): container finished" podID="400fc179-6148-4af6-b069-88f843600416" containerID="71d469b3c678d63ecb92fe3295e938187b187d4a1be0b0ba562a64b90f03e5f8" exitCode=0
Oct 31 00:54:20.384648 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.384559    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv" event={"ID":"400fc179-6148-4af6-b069-88f843600416","Type":"ContainerDied","Data":"71d469b3c678d63ecb92fe3295e938187b187d4a1be0b0ba562a64b90f03e5f8"}
Oct 31 00:54:20.384648 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.384606    4327 scope.go:122] "RemoveContainer" containerID="3504921cf5a6184bc5b78a5f156da93de8034956feeb12fd05501d0a3a920905"
Oct 31 00:54:20.656742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.654873    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:54:20.685701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.681268    4327 patch_prober.go:29] interesting pod/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 00:54:20.685701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.681336    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d" podUID="79f228f5-1947-426c-9e4d-4f300f43511e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 00:54:20.749703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.746017    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-workload-socket\") pod \"400fc179-6148-4af6-b069-88f843600416\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") "
Oct 31 00:54:20.749703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.746086    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/400fc179-6148-4af6-b069-88f843600416-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/400fc179-6148-4af6-b069-88f843600416-istiod-ca-cert\") pod \"400fc179-6148-4af6-b069-88f843600416\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") "
Oct 31 00:54:20.749703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.746126    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/400fc179-6148-4af6-b069-88f843600416-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/400fc179-6148-4af6-b069-88f843600416-istio-podinfo\") pod \"400fc179-6148-4af6-b069-88f843600416\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") "
Oct 31 00:54:20.749703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.746218    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/400fc179-6148-4af6-b069-88f843600416-istio-token\" (UniqueName: \"kubernetes.io/projected/400fc179-6148-4af6-b069-88f843600416-istio-token\") pod \"400fc179-6148-4af6-b069-88f843600416\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") "
Oct 31 00:54:20.749703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.746386    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-credential-socket\") pod \"400fc179-6148-4af6-b069-88f843600416\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") "
Oct 31 00:54:20.749703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.746460    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-istio-envoy\") pod \"400fc179-6148-4af6-b069-88f843600416\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") "
Oct 31 00:54:20.749703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.746554    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/400fc179-6148-4af6-b069-88f843600416-kube-api-access-bc6mm\" (UniqueName: \"kubernetes.io/projected/400fc179-6148-4af6-b069-88f843600416-kube-api-access-bc6mm\") pod \"400fc179-6148-4af6-b069-88f843600416\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") "
Oct 31 00:54:20.749703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.746600    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-workload-certs\") pod \"400fc179-6148-4af6-b069-88f843600416\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") "
Oct 31 00:54:20.749703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.746760    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-istio-data\") pod \"400fc179-6148-4af6-b069-88f843600416\" (UID: \"400fc179-6148-4af6-b069-88f843600416\") "
Oct 31 00:54:20.749703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.749003    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/400fc179-6148-4af6-b069-88f843600416-istiod-ca-cert" pod "400fc179-6148-4af6-b069-88f843600416" (UID: "400fc179-6148-4af6-b069-88f843600416"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:54:20.749703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.749130    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-istio-data" pod "400fc179-6148-4af6-b069-88f843600416" (UID: "400fc179-6148-4af6-b069-88f843600416"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:54:20.749703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.749331    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-workload-socket" pod "400fc179-6148-4af6-b069-88f843600416" (UID: "400fc179-6148-4af6-b069-88f843600416"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:54:20.780611 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.750800    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-credential-socket" pod "400fc179-6148-4af6-b069-88f843600416" (UID: "400fc179-6148-4af6-b069-88f843600416"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:54:20.780611 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.750986    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-workload-certs" pod "400fc179-6148-4af6-b069-88f843600416" (UID: "400fc179-6148-4af6-b069-88f843600416"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:54:20.780611 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.766915    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400fc179-6148-4af6-b069-88f843600416-kube-api-access-bc6mm" pod "400fc179-6148-4af6-b069-88f843600416" (UID: "400fc179-6148-4af6-b069-88f843600416"). InnerVolumeSpecName "kube-api-access-bc6mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:54:20.780611 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.780207    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/400fc179-6148-4af6-b069-88f843600416-istio-podinfo" pod "400fc179-6148-4af6-b069-88f843600416" (UID: "400fc179-6148-4af6-b069-88f843600416"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:54:20.781107 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.780742    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400fc179-6148-4af6-b069-88f843600416-istio-token" pod "400fc179-6148-4af6-b069-88f843600416" (UID: "400fc179-6148-4af6-b069-88f843600416"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:54:20.781178 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.781144    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-istio-envoy" pod "400fc179-6148-4af6-b069-88f843600416" (UID: "400fc179-6148-4af6-b069-88f843600416"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:54:20.852881 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.850193    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:20.852881 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.850231    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:20.852881 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.850247    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/400fc179-6148-4af6-b069-88f843600416-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:20.852881 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.850261    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/400fc179-6148-4af6-b069-88f843600416-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:20.852881 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.850274    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/400fc179-6148-4af6-b069-88f843600416-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:20.852881 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.850289    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:20.852881 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.850303    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:20.852881 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.850317    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc6mm\" (UniqueName: \"kubernetes.io/projected/400fc179-6148-4af6-b069-88f843600416-kube-api-access-bc6mm\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:20.852881 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:20.850329    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/400fc179-6148-4af6-b069-88f843600416-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:21.395123 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:21.395092    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"
Oct 31 00:54:21.395123 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:21.395114    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv" event={"ID":"400fc179-6148-4af6-b069-88f843600416","Type":"ContainerDied","Data":"f0adc72092361bcab543aaf741937164c13dfa5d29cef6863052e28e0d556ecb"}
Oct 31 00:54:21.395402 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:21.395186    4327 scope.go:122] "RemoveContainer" containerID="71d469b3c678d63ecb92fe3295e938187b187d4a1be0b0ba562a64b90f03e5f8"
Oct 31 00:54:21.471553 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:21.471522    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"]
Oct 31 00:54:21.477480 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:21.477451    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-68f412-openshift-default-fc857fc67-nccrv"]
Oct 31 00:54:21.653614 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:21.653536    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bbb902-65d446d445-znrq5"]
Oct 31 00:54:21.680781 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:21.680745    4327 patch_prober.go:29] interesting pod/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 00:54:21.680953 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:21.680797    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d" podUID="79f228f5-1947-426c-9e4d-4f300f43511e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 00:54:21.697549 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:21.697516    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6"]
Oct 31 00:54:22.111057 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:22.110985    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="400fc179-6148-4af6-b069-88f843600416" path="/var/lib/kubelet/pods/400fc179-6148-4af6-b069-88f843600416/volumes"
Oct 31 00:54:22.680523 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:22.680486    4327 patch_prober.go:29] interesting pod/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 00:54:22.680824 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:22.680789    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d" podUID="79f228f5-1947-426c-9e4d-4f300f43511e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 00:54:22.909840 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:22.909788    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"
Oct 31 00:54:23.020596 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:23.020511    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"
Oct 31 00:54:23.502438 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:23.502408    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"
Oct 31 00:54:23.681321 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:23.681289    4327 patch_prober.go:29] interesting pod/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 00:54:23.681501 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:23.681341    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d" podUID="79f228f5-1947-426c-9e4d-4f300f43511e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 00:54:24.686430 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:24.686400    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:54:24.686986 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:24.686767    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:54:24.687386 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:24.687362    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 00:54:26.052432 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:26.052399    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j"]
Oct 31 00:54:26.096059 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:26.096022    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst"]
Oct 31 00:54:26.654370 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:26.654323    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-bbb902-65d446d445-znrq5" podUID="59774227-9989-44ea-9e0d-ba76cd9e544d" containerName="payload-processing" containerID="cri-o://47bbe56a0b81774cca7270f01b7659e15f47b8be86b9f301f1e708d71379aefd" gracePeriod=25
Oct 31 00:54:26.700462 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:26.700395    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6" podUID="147e9302-09cd-4e2c-8a01-e31f299de460" containerName="payload-pre-processing" containerID="cri-o://07cac6a5aeb3ee7c387c616b7ea8f17654a62d99c293788cbedae6dc4d4ed1ac" gracePeriod=25
Oct 31 00:54:27.318983 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.318955    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bbb902-65d446d445-znrq5"
Oct 31 00:54:27.426651 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.426618    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59774227-9989-44ea-9e0d-ba76cd9e544d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59774227-9989-44ea-9e0d-ba76cd9e544d-plugins-config-volume\") pod \"59774227-9989-44ea-9e0d-ba76cd9e544d\" (UID: \"59774227-9989-44ea-9e0d-ba76cd9e544d\") "
Oct 31 00:54:27.427235 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.427208    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59774227-9989-44ea-9e0d-ba76cd9e544d-kube-api-access-sfn89\" (UniqueName: \"kubernetes.io/projected/59774227-9989-44ea-9e0d-ba76cd9e544d-kube-api-access-sfn89\") pod \"59774227-9989-44ea-9e0d-ba76cd9e544d\" (UID: \"59774227-9989-44ea-9e0d-ba76cd9e544d\") "
Oct 31 00:54:27.427454 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.427429    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59774227-9989-44ea-9e0d-ba76cd9e544d-plugins-config-volume" pod "59774227-9989-44ea-9e0d-ba76cd9e544d" (UID: "59774227-9989-44ea-9e0d-ba76cd9e544d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:54:27.428155 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.428132    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59774227-9989-44ea-9e0d-ba76cd9e544d-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:27.429867 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.429840    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6"
Oct 31 00:54:27.430236 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.430215    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59774227-9989-44ea-9e0d-ba76cd9e544d-kube-api-access-sfn89" pod "59774227-9989-44ea-9e0d-ba76cd9e544d" (UID: "59774227-9989-44ea-9e0d-ba76cd9e544d"). InnerVolumeSpecName "kube-api-access-sfn89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:54:27.529353 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.529321    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/147e9302-09cd-4e2c-8a01-e31f299de460-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147e9302-09cd-4e2c-8a01-e31f299de460-plugins-config-volume\") pod \"147e9302-09cd-4e2c-8a01-e31f299de460\" (UID: \"147e9302-09cd-4e2c-8a01-e31f299de460\") "
Oct 31 00:54:27.529856 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.529821    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147e9302-09cd-4e2c-8a01-e31f299de460-plugins-config-volume" pod "147e9302-09cd-4e2c-8a01-e31f299de460" (UID: "147e9302-09cd-4e2c-8a01-e31f299de460"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:54:27.530071 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.530023    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfn89\" (UniqueName: \"kubernetes.io/projected/59774227-9989-44ea-9e0d-ba76cd9e544d-kube-api-access-sfn89\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:27.530071 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.530049    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147e9302-09cd-4e2c-8a01-e31f299de460-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:27.531372 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.531343    4327 generic.go:356] "Generic (PLEG): container finished" podID="59774227-9989-44ea-9e0d-ba76cd9e544d" containerID="47bbe56a0b81774cca7270f01b7659e15f47b8be86b9f301f1e708d71379aefd" exitCode=0
Oct 31 00:54:27.531491 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.531394    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bbb902-65d446d445-znrq5"
Oct 31 00:54:27.531491 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.531426    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bbb902-65d446d445-znrq5" event={"ID":"59774227-9989-44ea-9e0d-ba76cd9e544d","Type":"ContainerDied","Data":"47bbe56a0b81774cca7270f01b7659e15f47b8be86b9f301f1e708d71379aefd"}
Oct 31 00:54:27.531605 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.531507    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bbb902-65d446d445-znrq5" event={"ID":"59774227-9989-44ea-9e0d-ba76cd9e544d","Type":"ContainerDied","Data":"8c6e4fd528118e2430ff55981d77e77a8c2a9293acf8aab0b27be13c0f10315b"}
Oct 31 00:54:27.531605 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.531569    4327 scope.go:122] "RemoveContainer" containerID="47bbe56a0b81774cca7270f01b7659e15f47b8be86b9f301f1e708d71379aefd"
Oct 31 00:54:27.533325 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.533304    4327 generic.go:356] "Generic (PLEG): container finished" podID="147e9302-09cd-4e2c-8a01-e31f299de460" containerID="07cac6a5aeb3ee7c387c616b7ea8f17654a62d99c293788cbedae6dc4d4ed1ac" exitCode=0
Oct 31 00:54:27.533490 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.533347    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6"
Oct 31 00:54:27.533775 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.533378    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6" event={"ID":"147e9302-09cd-4e2c-8a01-e31f299de460","Type":"ContainerDied","Data":"07cac6a5aeb3ee7c387c616b7ea8f17654a62d99c293788cbedae6dc4d4ed1ac"}
Oct 31 00:54:27.533871 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.533790    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6" event={"ID":"147e9302-09cd-4e2c-8a01-e31f299de460","Type":"ContainerDied","Data":"726db652ae7b46e8ab3d6285976ba814bc4af4f9ee2ab9b71779b543e59c2721"}
Oct 31 00:54:27.544547 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.544530    4327 scope.go:122] "RemoveContainer" containerID="47bbe56a0b81774cca7270f01b7659e15f47b8be86b9f301f1e708d71379aefd"
Oct 31 00:54:27.544879 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:54:27.544859    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47bbe56a0b81774cca7270f01b7659e15f47b8be86b9f301f1e708d71379aefd\": container with ID starting with 47bbe56a0b81774cca7270f01b7659e15f47b8be86b9f301f1e708d71379aefd not found: ID does not exist" containerID="47bbe56a0b81774cca7270f01b7659e15f47b8be86b9f301f1e708d71379aefd"
Oct 31 00:54:27.544954 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.544885    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47bbe56a0b81774cca7270f01b7659e15f47b8be86b9f301f1e708d71379aefd"} err="failed to get container status \"47bbe56a0b81774cca7270f01b7659e15f47b8be86b9f301f1e708d71379aefd\": rpc error: code = NotFound desc = could not find container \"47bbe56a0b81774cca7270f01b7659e15f47b8be86b9f301f1e708d71379aefd\": container with ID starting with 47bbe56a0b81774cca7270f01b7659e15f47b8be86b9f301f1e708d71379aefd not found: ID does not exist"
Oct 31 00:54:27.544954 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.544901    4327 scope.go:122] "RemoveContainer" containerID="07cac6a5aeb3ee7c387c616b7ea8f17654a62d99c293788cbedae6dc4d4ed1ac"
Oct 31 00:54:27.555242 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.555210    4327 scope.go:122] "RemoveContainer" containerID="07cac6a5aeb3ee7c387c616b7ea8f17654a62d99c293788cbedae6dc4d4ed1ac"
Oct 31 00:54:27.555785 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:54:27.555754    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07cac6a5aeb3ee7c387c616b7ea8f17654a62d99c293788cbedae6dc4d4ed1ac\": container with ID starting with 07cac6a5aeb3ee7c387c616b7ea8f17654a62d99c293788cbedae6dc4d4ed1ac not found: ID does not exist" containerID="07cac6a5aeb3ee7c387c616b7ea8f17654a62d99c293788cbedae6dc4d4ed1ac"
Oct 31 00:54:27.555867 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.555796    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07cac6a5aeb3ee7c387c616b7ea8f17654a62d99c293788cbedae6dc4d4ed1ac"} err="failed to get container status \"07cac6a5aeb3ee7c387c616b7ea8f17654a62d99c293788cbedae6dc4d4ed1ac\": rpc error: code = NotFound desc = could not find container \"07cac6a5aeb3ee7c387c616b7ea8f17654a62d99c293788cbedae6dc4d4ed1ac\": container with ID starting with 07cac6a5aeb3ee7c387c616b7ea8f17654a62d99c293788cbedae6dc4d4ed1ac not found: ID does not exist"
Oct 31 00:54:27.602494 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.602458    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bbb902-65d446d445-znrq5"]
Oct 31 00:54:27.608160 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.608137    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bbb902-65d446d445-znrq5"]
Oct 31 00:54:27.630644 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.630620    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147e9302-09cd-4e2c-8a01-e31f299de460-kube-api-access-262tw\" (UniqueName: \"kubernetes.io/projected/147e9302-09cd-4e2c-8a01-e31f299de460-kube-api-access-262tw\") pod \"147e9302-09cd-4e2c-8a01-e31f299de460\" (UID: \"147e9302-09cd-4e2c-8a01-e31f299de460\") "
Oct 31 00:54:27.632777 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.632757    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147e9302-09cd-4e2c-8a01-e31f299de460-kube-api-access-262tw" pod "147e9302-09cd-4e2c-8a01-e31f299de460" (UID: "147e9302-09cd-4e2c-8a01-e31f299de460"). InnerVolumeSpecName "kube-api-access-262tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:54:27.732373 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.732339    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-262tw\" (UniqueName: \"kubernetes.io/projected/147e9302-09cd-4e2c-8a01-e31f299de460-kube-api-access-262tw\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:27.881517 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.881485    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6"]
Oct 31 00:54:27.885331 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.885310    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bbb902-74dbf5bf87-69lb6"]
Oct 31 00:54:27.925647 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.925571    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="147e9302-09cd-4e2c-8a01-e31f299de460" path="/var/lib/kubelet/pods/147e9302-09cd-4e2c-8a01-e31f299de460/volumes"
Oct 31 00:54:27.926114 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:27.926097    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59774227-9989-44ea-9e0d-ba76cd9e544d" path="/var/lib/kubelet/pods/59774227-9989-44ea-9e0d-ba76cd9e544d/volumes"
Oct 31 00:54:31.053435 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.053395    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j" podUID="dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e" containerName="payload-processing" containerID="cri-o://83583a4d2cdcb97f23a0c0345111114a0b538d4962d19029e30949ee64123dae" gracePeriod=25
Oct 31 00:54:31.096648 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.096610    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst" podUID="f17abb15-a890-44cf-a93e-fe9580905cd4" containerName="payload-pre-processing" containerID="cri-o://cdfa86bf177cf3d1081f2dae218f55b0a8671c25ae7e463f6cd440ad4ffa5019" gracePeriod=25
Oct 31 00:54:31.404249 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.404216    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j"
Oct 31 00:54:31.438880 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.438852    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst"
Oct 31 00:54:31.509945 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.508278    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f17abb15-a890-44cf-a93e-fe9580905cd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f17abb15-a890-44cf-a93e-fe9580905cd4-plugins-config-volume\") pod \"f17abb15-a890-44cf-a93e-fe9580905cd4\" (UID: \"f17abb15-a890-44cf-a93e-fe9580905cd4\") "
Oct 31 00:54:31.509945 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.508403    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e-kube-api-access-vclh5\" (UniqueName: \"kubernetes.io/projected/dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e-kube-api-access-vclh5\") pod \"dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e\" (UID: \"dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e\") "
Oct 31 00:54:31.509945 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.508607    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f17abb15-a890-44cf-a93e-fe9580905cd4-kube-api-access-gks8z\" (UniqueName: \"kubernetes.io/projected/f17abb15-a890-44cf-a93e-fe9580905cd4-kube-api-access-gks8z\") pod \"f17abb15-a890-44cf-a93e-fe9580905cd4\" (UID: \"f17abb15-a890-44cf-a93e-fe9580905cd4\") "
Oct 31 00:54:31.520791 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.511756    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f17abb15-a890-44cf-a93e-fe9580905cd4-plugins-config-volume" pod "f17abb15-a890-44cf-a93e-fe9580905cd4" (UID: "f17abb15-a890-44cf-a93e-fe9580905cd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:54:31.520791 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.512221    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e-plugins-config-volume" pod "dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e" (UID: "dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:54:31.535336 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.508646    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e-plugins-config-volume\") pod \"dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e\" (UID: \"dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e\") "
Oct 31 00:54:31.535336 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.531238    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f17abb15-a890-44cf-a93e-fe9580905cd4-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:31.535336 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.531267    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:31.539242 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.539188    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f17abb15-a890-44cf-a93e-fe9580905cd4-kube-api-access-gks8z" pod "f17abb15-a890-44cf-a93e-fe9580905cd4" (UID: "f17abb15-a890-44cf-a93e-fe9580905cd4"). InnerVolumeSpecName "kube-api-access-gks8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:54:31.549762 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.541605    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e-kube-api-access-vclh5" pod "dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e" (UID: "dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e"). InnerVolumeSpecName "kube-api-access-vclh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:54:31.632938 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.632896    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vclh5\" (UniqueName: \"kubernetes.io/projected/dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e-kube-api-access-vclh5\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:31.632938 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.632931    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gks8z\" (UniqueName: \"kubernetes.io/projected/f17abb15-a890-44cf-a93e-fe9580905cd4-kube-api-access-gks8z\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:31.635991 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.635956    4327 generic.go:356] "Generic (PLEG): container finished" podID="dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e" containerID="83583a4d2cdcb97f23a0c0345111114a0b538d4962d19029e30949ee64123dae" exitCode=0
Oct 31 00:54:31.636140 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.636009    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j"
Oct 31 00:54:31.636140 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.636042    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j" event={"ID":"dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e","Type":"ContainerDied","Data":"83583a4d2cdcb97f23a0c0345111114a0b538d4962d19029e30949ee64123dae"}
Oct 31 00:54:31.636140 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.636110    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j" event={"ID":"dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e","Type":"ContainerDied","Data":"f76c2ec98761ac2d04501cff5de49d9743a0799f1d7cd6f2cde429c4e29fe041"}
Oct 31 00:54:31.636320 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.636143    4327 scope.go:122] "RemoveContainer" containerID="83583a4d2cdcb97f23a0c0345111114a0b538d4962d19029e30949ee64123dae"
Oct 31 00:54:31.638513 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.638484    4327 generic.go:356] "Generic (PLEG): container finished" podID="f17abb15-a890-44cf-a93e-fe9580905cd4" containerID="cdfa86bf177cf3d1081f2dae218f55b0a8671c25ae7e463f6cd440ad4ffa5019" exitCode=0
Oct 31 00:54:31.638667 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.638648    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst" event={"ID":"f17abb15-a890-44cf-a93e-fe9580905cd4","Type":"ContainerDied","Data":"cdfa86bf177cf3d1081f2dae218f55b0a8671c25ae7e463f6cd440ad4ffa5019"}
Oct 31 00:54:31.638753 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.638741    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst" event={"ID":"f17abb15-a890-44cf-a93e-fe9580905cd4","Type":"ContainerDied","Data":"0da1fd2e177bf32ce1057a2d0c56d9bcbde16487645b1e8ba6ab9438bfb12da8"}
Oct 31 00:54:31.638836 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.638817    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst"
Oct 31 00:54:31.655539 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.655518    4327 scope.go:122] "RemoveContainer" containerID="83583a4d2cdcb97f23a0c0345111114a0b538d4962d19029e30949ee64123dae"
Oct 31 00:54:31.655900 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:54:31.655874    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83583a4d2cdcb97f23a0c0345111114a0b538d4962d19029e30949ee64123dae\": container with ID starting with 83583a4d2cdcb97f23a0c0345111114a0b538d4962d19029e30949ee64123dae not found: ID does not exist" containerID="83583a4d2cdcb97f23a0c0345111114a0b538d4962d19029e30949ee64123dae"
Oct 31 00:54:31.656033 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.655906    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83583a4d2cdcb97f23a0c0345111114a0b538d4962d19029e30949ee64123dae"} err="failed to get container status \"83583a4d2cdcb97f23a0c0345111114a0b538d4962d19029e30949ee64123dae\": rpc error: code = NotFound desc = could not find container \"83583a4d2cdcb97f23a0c0345111114a0b538d4962d19029e30949ee64123dae\": container with ID starting with 83583a4d2cdcb97f23a0c0345111114a0b538d4962d19029e30949ee64123dae not found: ID does not exist"
Oct 31 00:54:31.656033 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.655923    4327 scope.go:122] "RemoveContainer" containerID="cdfa86bf177cf3d1081f2dae218f55b0a8671c25ae7e463f6cd440ad4ffa5019"
Oct 31 00:54:31.666952 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.666931    4327 scope.go:122] "RemoveContainer" containerID="cdfa86bf177cf3d1081f2dae218f55b0a8671c25ae7e463f6cd440ad4ffa5019"
Oct 31 00:54:31.667227 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:54:31.667205    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdfa86bf177cf3d1081f2dae218f55b0a8671c25ae7e463f6cd440ad4ffa5019\": container with ID starting with cdfa86bf177cf3d1081f2dae218f55b0a8671c25ae7e463f6cd440ad4ffa5019 not found: ID does not exist" containerID="cdfa86bf177cf3d1081f2dae218f55b0a8671c25ae7e463f6cd440ad4ffa5019"
Oct 31 00:54:31.667281 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.667235    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdfa86bf177cf3d1081f2dae218f55b0a8671c25ae7e463f6cd440ad4ffa5019"} err="failed to get container status \"cdfa86bf177cf3d1081f2dae218f55b0a8671c25ae7e463f6cd440ad4ffa5019\": rpc error: code = NotFound desc = could not find container \"cdfa86bf177cf3d1081f2dae218f55b0a8671c25ae7e463f6cd440ad4ffa5019\": container with ID starting with cdfa86bf177cf3d1081f2dae218f55b0a8671c25ae7e463f6cd440ad4ffa5019 not found: ID does not exist"
Oct 31 00:54:31.706481 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.706446    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j"]
Oct 31 00:54:31.710264 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.710244    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-82d887-76f55bc4f5-p7m4j"]
Oct 31 00:54:31.745943 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.745901    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst"]
Oct 31 00:54:31.749708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.749655    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-82d887-567965dcbc-7zpst"]
Oct 31 00:54:31.924373 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.924282    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e" path="/var/lib/kubelet/pods/dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e/volumes"
Oct 31 00:54:31.924908 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:31.924885    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f17abb15-a890-44cf-a93e-fe9580905cd4" path="/var/lib/kubelet/pods/f17abb15-a890-44cf-a93e-fe9580905cd4/volumes"
Oct 31 00:54:41.950847 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:41.950812    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"]
Oct 31 00:54:41.951700 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:41.951668    4327 cpu_manager.go:414] "containerMap: removing container" podUID="400fc179-6148-4af6-b069-88f843600416" containerName="istio-proxy"
Oct 31 00:54:41.951755 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:41.951702    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="400fc179-6148-4af6-b069-88f843600416" containerName="istio-proxy"
Oct 31 00:54:41.951755 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:41.951716    4327 cpu_manager.go:414] "containerMap: removing container" podUID="f17abb15-a890-44cf-a93e-fe9580905cd4" containerName="payload-pre-processing"
Oct 31 00:54:41.951755 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:41.951721    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f17abb15-a890-44cf-a93e-fe9580905cd4" containerName="payload-pre-processing"
Oct 31 00:54:41.951755 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:41.951735    4327 cpu_manager.go:414] "containerMap: removing container" podUID="dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e" containerName="payload-processing"
Oct 31 00:54:41.951755 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:41.951740    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e" containerName="payload-processing"
Oct 31 00:54:41.951926 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:41.951786    4327 cpu_manager.go:414] "containerMap: removing container" podUID="147e9302-09cd-4e2c-8a01-e31f299de460" containerName="payload-pre-processing"
Oct 31 00:54:41.951926 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:41.951792    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147e9302-09cd-4e2c-8a01-e31f299de460" containerName="payload-pre-processing"
Oct 31 00:54:41.951926 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:41.951809    4327 cpu_manager.go:414] "containerMap: removing container" podUID="59774227-9989-44ea-9e0d-ba76cd9e544d" containerName="payload-processing"
Oct 31 00:54:41.951926 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:41.951814    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59774227-9989-44ea-9e0d-ba76cd9e544d" containerName="payload-processing"
Oct 31 00:54:41.952046 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:41.951929    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="147e9302-09cd-4e2c-8a01-e31f299de460" containerName="payload-pre-processing"
Oct 31 00:54:41.952046 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:41.951938    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="59774227-9989-44ea-9e0d-ba76cd9e544d" containerName="payload-processing"
Oct 31 00:54:41.952046 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:41.951946    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="400fc179-6148-4af6-b069-88f843600416" containerName="istio-proxy"
Oct 31 00:54:41.952046 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:41.951954    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="400fc179-6148-4af6-b069-88f843600416" containerName="istio-proxy"
Oct 31 00:54:41.952046 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:41.951961    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="f17abb15-a890-44cf-a93e-fe9580905cd4" containerName="payload-pre-processing"
Oct 31 00:54:41.952046 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:41.951967    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc6bec5c-5bf7-4e9c-b33b-caf337aeb76e" containerName="payload-processing"
Oct 31 00:54:41.957360 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:41.957339    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"
Oct 31 00:54:41.979136 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:41.979103    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"]
Oct 31 00:54:41.984525 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:41.984497    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c6d97d\"/\"default-dockercfg-zvn6x\""
Oct 31 00:54:41.984747 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:41.984660    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c6d97d\"/\"test-model-c6d97d-kserve-self-signed-certs\""
Oct 31 00:54:41.989597 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:41.989569    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c6d97d\"/\"openshift-service-ca.crt\""
Oct 31 00:54:41.989767 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:41.989640    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c6d97d\"/\"kube-root-ca.crt\""
Oct 31 00:54:42.017174 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:42.017137    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-tmp-dir\") pod \"test-model-c6d97d-kserve-7585856c57-x56f8\" (UID: \"a175adcd-0c7a-4572-940c-96eca0ee58d5\") " pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"
Oct 31 00:54:42.017366 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:42.017189    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a175adcd-0c7a-4572-940c-96eca0ee58d5-tls-certs\") pod \"test-model-c6d97d-kserve-7585856c57-x56f8\" (UID: \"a175adcd-0c7a-4572-940c-96eca0ee58d5\") " pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"
Oct 31 00:54:42.017366 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:42.017318    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-home\") pod \"test-model-c6d97d-kserve-7585856c57-x56f8\" (UID: \"a175adcd-0c7a-4572-940c-96eca0ee58d5\") " pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"
Oct 31 00:54:42.017458 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:42.017444    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-dshm\") pod \"test-model-c6d97d-kserve-7585856c57-x56f8\" (UID: \"a175adcd-0c7a-4572-940c-96eca0ee58d5\") " pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"
Oct 31 00:54:42.017598 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:42.017582    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-model-cache\") pod \"test-model-c6d97d-kserve-7585856c57-x56f8\" (UID: \"a175adcd-0c7a-4572-940c-96eca0ee58d5\") " pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"
Oct 31 00:54:42.017712 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:42.017694    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp4gz\" (UniqueName: \"kubernetes.io/projected/a175adcd-0c7a-4572-940c-96eca0ee58d5-kube-api-access-dp4gz\") pod \"test-model-c6d97d-kserve-7585856c57-x56f8\" (UID: \"a175adcd-0c7a-4572-940c-96eca0ee58d5\") " pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"
Oct 31 00:54:42.119338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:42.119289    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-tmp-dir\") pod \"test-model-c6d97d-kserve-7585856c57-x56f8\" (UID: \"a175adcd-0c7a-4572-940c-96eca0ee58d5\") " pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"
Oct 31 00:54:42.119531 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:42.119357    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a175adcd-0c7a-4572-940c-96eca0ee58d5-tls-certs\") pod \"test-model-c6d97d-kserve-7585856c57-x56f8\" (UID: \"a175adcd-0c7a-4572-940c-96eca0ee58d5\") " pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"
Oct 31 00:54:42.119531 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:42.119393    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-home\") pod \"test-model-c6d97d-kserve-7585856c57-x56f8\" (UID: \"a175adcd-0c7a-4572-940c-96eca0ee58d5\") " pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"
Oct 31 00:54:42.119531 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:42.119500    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-dshm\") pod \"test-model-c6d97d-kserve-7585856c57-x56f8\" (UID: \"a175adcd-0c7a-4572-940c-96eca0ee58d5\") " pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"
Oct 31 00:54:42.119759 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:42.119657    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-model-cache\") pod \"test-model-c6d97d-kserve-7585856c57-x56f8\" (UID: \"a175adcd-0c7a-4572-940c-96eca0ee58d5\") " pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"
Oct 31 00:54:42.119835 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:42.119791    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-home\") pod \"test-model-c6d97d-kserve-7585856c57-x56f8\" (UID: \"a175adcd-0c7a-4572-940c-96eca0ee58d5\") " pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"
Oct 31 00:54:42.119835 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:42.119792    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-tmp-dir\") pod \"test-model-c6d97d-kserve-7585856c57-x56f8\" (UID: \"a175adcd-0c7a-4572-940c-96eca0ee58d5\") " pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"
Oct 31 00:54:42.119952 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:42.119887    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp4gz\" (UniqueName: \"kubernetes.io/projected/a175adcd-0c7a-4572-940c-96eca0ee58d5-kube-api-access-dp4gz\") pod \"test-model-c6d97d-kserve-7585856c57-x56f8\" (UID: \"a175adcd-0c7a-4572-940c-96eca0ee58d5\") " pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"
Oct 31 00:54:42.120122 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:42.120101    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-model-cache\") pod \"test-model-c6d97d-kserve-7585856c57-x56f8\" (UID: \"a175adcd-0c7a-4572-940c-96eca0ee58d5\") " pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"
Oct 31 00:54:42.121830 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:42.121807    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-dshm\") pod \"test-model-c6d97d-kserve-7585856c57-x56f8\" (UID: \"a175adcd-0c7a-4572-940c-96eca0ee58d5\") " pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"
Oct 31 00:54:42.122092 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:42.122075    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a175adcd-0c7a-4572-940c-96eca0ee58d5-tls-certs\") pod \"test-model-c6d97d-kserve-7585856c57-x56f8\" (UID: \"a175adcd-0c7a-4572-940c-96eca0ee58d5\") " pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"
Oct 31 00:54:42.146760 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:42.146729    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp4gz\" (UniqueName: \"kubernetes.io/projected/a175adcd-0c7a-4572-940c-96eca0ee58d5-kube-api-access-dp4gz\") pod \"test-model-c6d97d-kserve-7585856c57-x56f8\" (UID: \"a175adcd-0c7a-4572-940c-96eca0ee58d5\") " pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"
Oct 31 00:54:42.297763 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:42.286442    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"
Oct 31 00:54:42.879581 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:42.879553    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"]
Oct 31 00:54:42.881794 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:54:42.881765    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda175adcd_0c7a_4572_940c_96eca0ee58d5.slice/crio-5a39d6acf70ccc76f63a5fa3547c80768edd3ec033970f40ba9f66f61365ce2b WatchSource:0}: Error finding container 5a39d6acf70ccc76f63a5fa3547c80768edd3ec033970f40ba9f66f61365ce2b: Status 404 returned error can't find the container with id 5a39d6acf70ccc76f63a5fa3547c80768edd3ec033970f40ba9f66f61365ce2b
Oct 31 00:54:43.505343 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.505311    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"]
Oct 31 00:54:43.506223 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.506198    4327 cpu_manager.go:414] "containerMap: removing container" podUID="400fc179-6148-4af6-b069-88f843600416" containerName="istio-proxy"
Oct 31 00:54:43.506223 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.506221    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="400fc179-6148-4af6-b069-88f843600416" containerName="istio-proxy"
Oct 31 00:54:43.511347 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.511302    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"
Oct 31 00:54:43.536039 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.536005    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"]
Oct 31 00:54:43.542310 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.541954    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fd1dfc\"/\"openshift-service-ca.crt\""
Oct 31 00:54:43.542310 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.541976    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fd1dfc\"/\"kube-root-ca.crt\""
Oct 31 00:54:43.542310 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.542082    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fd1dfc\"/\"default-dockercfg-s5wbr\""
Oct 31 00:54:43.542310 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.542149    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fd1dfc\"/\"test-model-fd1dfc-kserve-self-signed-certs\""
Oct 31 00:54:43.564504 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.564474    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc7r9\" (UniqueName: \"kubernetes.io/projected/734185a4-0ff7-4580-abb7-865dee21b8aa-kube-api-access-nc7r9\") pod \"test-model-fd1dfc-kserve-5d759978c9-74txf\" (UID: \"734185a4-0ff7-4580-abb7-865dee21b8aa\") " pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"
Oct 31 00:54:43.564714 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.564521    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/734185a4-0ff7-4580-abb7-865dee21b8aa-tls-certs\") pod \"test-model-fd1dfc-kserve-5d759978c9-74txf\" (UID: \"734185a4-0ff7-4580-abb7-865dee21b8aa\") " pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"
Oct 31 00:54:43.564714 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.564548    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-home\") pod \"test-model-fd1dfc-kserve-5d759978c9-74txf\" (UID: \"734185a4-0ff7-4580-abb7-865dee21b8aa\") " pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"
Oct 31 00:54:43.564851 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.564749    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-dshm\") pod \"test-model-fd1dfc-kserve-5d759978c9-74txf\" (UID: \"734185a4-0ff7-4580-abb7-865dee21b8aa\") " pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"
Oct 31 00:54:43.564934 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.564915    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-tmp-dir\") pod \"test-model-fd1dfc-kserve-5d759978c9-74txf\" (UID: \"734185a4-0ff7-4580-abb7-865dee21b8aa\") " pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"
Oct 31 00:54:43.565115 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.565095    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-model-cache\") pod \"test-model-fd1dfc-kserve-5d759978c9-74txf\" (UID: \"734185a4-0ff7-4580-abb7-865dee21b8aa\") " pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"
Oct 31 00:54:43.667129 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.667092    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-model-cache\") pod \"test-model-fd1dfc-kserve-5d759978c9-74txf\" (UID: \"734185a4-0ff7-4580-abb7-865dee21b8aa\") " pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"
Oct 31 00:54:43.667321 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.667210    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc7r9\" (UniqueName: \"kubernetes.io/projected/734185a4-0ff7-4580-abb7-865dee21b8aa-kube-api-access-nc7r9\") pod \"test-model-fd1dfc-kserve-5d759978c9-74txf\" (UID: \"734185a4-0ff7-4580-abb7-865dee21b8aa\") " pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"
Oct 31 00:54:43.667321 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.667244    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/734185a4-0ff7-4580-abb7-865dee21b8aa-tls-certs\") pod \"test-model-fd1dfc-kserve-5d759978c9-74txf\" (UID: \"734185a4-0ff7-4580-abb7-865dee21b8aa\") " pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"
Oct 31 00:54:43.667321 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.667267    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-home\") pod \"test-model-fd1dfc-kserve-5d759978c9-74txf\" (UID: \"734185a4-0ff7-4580-abb7-865dee21b8aa\") " pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"
Oct 31 00:54:43.667493 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.667359    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-dshm\") pod \"test-model-fd1dfc-kserve-5d759978c9-74txf\" (UID: \"734185a4-0ff7-4580-abb7-865dee21b8aa\") " pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"
Oct 31 00:54:43.667560 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.667500    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-tmp-dir\") pod \"test-model-fd1dfc-kserve-5d759978c9-74txf\" (UID: \"734185a4-0ff7-4580-abb7-865dee21b8aa\") " pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"
Oct 31 00:54:43.667560 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.667517    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-model-cache\") pod \"test-model-fd1dfc-kserve-5d759978c9-74txf\" (UID: \"734185a4-0ff7-4580-abb7-865dee21b8aa\") " pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"
Oct 31 00:54:43.667725 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.667668    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-home\") pod \"test-model-fd1dfc-kserve-5d759978c9-74txf\" (UID: \"734185a4-0ff7-4580-abb7-865dee21b8aa\") " pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"
Oct 31 00:54:43.667912 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.667885    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-tmp-dir\") pod \"test-model-fd1dfc-kserve-5d759978c9-74txf\" (UID: \"734185a4-0ff7-4580-abb7-865dee21b8aa\") " pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"
Oct 31 00:54:43.669794 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.669767    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-dshm\") pod \"test-model-fd1dfc-kserve-5d759978c9-74txf\" (UID: \"734185a4-0ff7-4580-abb7-865dee21b8aa\") " pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"
Oct 31 00:54:43.669902 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.669885    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/734185a4-0ff7-4580-abb7-865dee21b8aa-tls-certs\") pod \"test-model-fd1dfc-kserve-5d759978c9-74txf\" (UID: \"734185a4-0ff7-4580-abb7-865dee21b8aa\") " pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"
Oct 31 00:54:43.725990 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.725947    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc7r9\" (UniqueName: \"kubernetes.io/projected/734185a4-0ff7-4580-abb7-865dee21b8aa-kube-api-access-nc7r9\") pod \"test-model-fd1dfc-kserve-5d759978c9-74txf\" (UID: \"734185a4-0ff7-4580-abb7-865dee21b8aa\") " pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"
Oct 31 00:54:43.743071 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.743034    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8" event={"ID":"a175adcd-0c7a-4572-940c-96eca0ee58d5","Type":"ContainerStarted","Data":"c2fe3a1404b72d3d5d21d27039238d846b03c8f1d1b77b416478d63d60df6dd3"}
Oct 31 00:54:43.743071 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.743077    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8" event={"ID":"a175adcd-0c7a-4572-940c-96eca0ee58d5","Type":"ContainerStarted","Data":"5a39d6acf70ccc76f63a5fa3547c80768edd3ec033970f40ba9f66f61365ce2b"}
Oct 31 00:54:43.812738 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.812609    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8" podStartSLOduration=2.812593718 podStartE2EDuration="2.812593718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:43.810038666 +0000 UTC m=+2144.032074209" watchObservedRunningTime="2026-10-31 00:54:43.812593718 +0000 UTC m=+2144.034629323"
Oct 31 00:54:43.831745 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:43.831708    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"
Oct 31 00:54:44.237790 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:44.235867    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"]
Oct 31 00:54:44.304668 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:44.304634    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"
Oct 31 00:54:44.772063 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:44.771981    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf" event={"ID":"734185a4-0ff7-4580-abb7-865dee21b8aa","Type":"ContainerStarted","Data":"1186698632d7057827622a69c91745082dee4d6767b902c74364eae18ed3a872"}
Oct 31 00:54:44.772063 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:44.772024    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf" event={"ID":"734185a4-0ff7-4580-abb7-865dee21b8aa","Type":"ContainerStarted","Data":"c2ebfc18c5d9a7285e9497c4e4b2235d12c454a218a4fc68515d3acc49ee36bb"}
Oct 31 00:54:44.815839 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:44.815787    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf" podStartSLOduration=1.815765764 podStartE2EDuration="1.815765764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:44.814097213 +0000 UTC m=+2145.036132847" watchObservedRunningTime="2026-10-31 00:54:44.815765764 +0000 UTC m=+2145.037801306"
Oct 31 00:54:44.842601 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:44.841852    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"
Oct 31 00:54:45.431016 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:54:45.430221    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:54:50.034397 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:50.034353    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"]
Oct 31 00:54:50.034931 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:50.034576    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7" podUID="c5c6a0bf-623b-41e0-9c60-4a3fa55e8770" containerName="istio-proxy" containerID="cri-o://9a185e9c58176b62d3a4e9195aa9e1607687144d3bbc1d0ba299ace77be5fb96" gracePeriod=30
Oct 31 00:54:50.478210 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:50.478174    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"]
Oct 31 00:54:50.478408 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:50.478378    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9" podUID="1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c" containerName="istio-proxy" containerID="cri-o://c5165caf7853cd703ac494fc85f1a4f5dd18a0cda7d66b1e4d497148e5135681" gracePeriod=30
Oct 31 00:54:50.905622 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:50.905588    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"]
Oct 31 00:54:50.912135 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:50.912105    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:50.947079 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:50.947044    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"]
Oct 31 00:54:50.961139 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:50.961111    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-01a117-openshift-default-dockercfg-ltlbc\""
Oct 31 00:54:51.239754 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.171956    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-token\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.239754 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.172155    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-workload-certs\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.239754 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.172336    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-credential-socket\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.239754 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.172414    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istiod-ca-cert\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.239754 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.172458    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-envoy\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.239754 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.172527    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-workload-socket\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.239754 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.172563    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-data\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.239754 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.172590    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgdzb\" (UniqueName: \"kubernetes.io/projected/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-kube-api-access-sgdzb\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.239754 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.172642    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-podinfo\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.295721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.279384    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istiod-ca-cert\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.295721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.279468    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-envoy\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.295721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.279531    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-workload-socket\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.295721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.279558    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-data\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.295721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.279581    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgdzb\" (UniqueName: \"kubernetes.io/projected/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-kube-api-access-sgdzb\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.295721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.279624    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-podinfo\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.295721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.279671    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-token\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.295721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.279836    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-workload-certs\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.295721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.279999    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-credential-socket\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.295721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.280763    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-credential-socket\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.295721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.281512    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istiod-ca-cert\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.295721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.295184    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-workload-socket\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.295721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.295490    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-data\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.306866 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.305057    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-workload-certs\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.317471 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.317218    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-podinfo\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.340150 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.340116    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgdzb\" (UniqueName: \"kubernetes.io/projected/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-kube-api-access-sgdzb\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.349753 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.341260    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-envoy\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.354292 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.354218    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-token\") pod \"e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") " pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.537056 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.536977    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:51.875185 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.873879    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"]
Oct 31 00:54:51.885981 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:54:51.885944    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7a3f22_3f9e_46a4_bed7_5591bdbcb3f3.slice/crio-c78eb8c31ad35fbfb7e97b63e388753bb152e14c3fe6a29708cb6c4e75e74265 WatchSource:0}: Error finding container c78eb8c31ad35fbfb7e97b63e388753bb152e14c3fe6a29708cb6c4e75e74265: Status 404 returned error can't find the container with id c78eb8c31ad35fbfb7e97b63e388753bb152e14c3fe6a29708cb6c4e75e74265
Oct 31 00:54:51.890421 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.890383    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:54:51.890535 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.890457    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:54:51.890535 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:51.890487    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:54:52.111861 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:52.111808    4327 patch_prober.go:29] interesting pod/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:54:52.112055 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:52.111884    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk" podUID="242be892-3b92-44cb-a8ef-17f8ac581b4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:54:52.291581 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:52.291480    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"
Oct 31 00:54:52.307596 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:52.307248    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"
Oct 31 00:54:52.861692 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:52.861646    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz" event={"ID":"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3","Type":"ContainerStarted","Data":"36af665e2edb2165b20f033f90248a1eb21445866c3f379f14a0d09c59861dcc"}
Oct 31 00:54:52.861898 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:52.861751    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz" event={"ID":"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3","Type":"ContainerStarted","Data":"c78eb8c31ad35fbfb7e97b63e388753bb152e14c3fe6a29708cb6c4e75e74265"}
Oct 31 00:54:52.874095 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:52.874068    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"
Oct 31 00:54:52.944510 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:52.944442    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz" podStartSLOduration=2.944425895 podStartE2EDuration="2.944425895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:52.930312537 +0000 UTC m=+2153.152348078" watchObservedRunningTime="2026-10-31 00:54:52.944425895 +0000 UTC m=+2153.166461510"
Oct 31 00:54:53.544703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:53.541498    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:53.697707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:53.697193    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:53.832762 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:53.832625    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"
Oct 31 00:54:53.846516 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:53.846487    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"
Oct 31 00:54:53.871170 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:53.870965    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:53.872318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:53.872293    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:54:53.883430 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:53.883400    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"
Oct 31 00:54:55.361631 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.361606    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:54:55.501606 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.501574    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-data\") pod \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") "
Oct 31 00:54:55.501942 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.501918    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-token\" (UniqueName: \"kubernetes.io/projected/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-token\") pod \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") "
Oct 31 00:54:55.502662 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.502055    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-data" pod "c5c6a0bf-623b-41e0-9c60-4a3fa55e8770" (UID: "c5c6a0bf-623b-41e0-9c60-4a3fa55e8770"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:54:55.502936 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.502916    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-workload-certs\") pod \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") "
Oct 31 00:54:55.503346 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.503327    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-envoy\") pod \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") "
Oct 31 00:54:55.503546 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.503242    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-workload-certs" pod "c5c6a0bf-623b-41e0-9c60-4a3fa55e8770" (UID: "c5c6a0bf-623b-41e0-9c60-4a3fa55e8770"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:54:55.503913 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.503569    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-kube-api-access-lrswv\" (UniqueName: \"kubernetes.io/projected/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-kube-api-access-lrswv\") pod \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") "
Oct 31 00:54:55.504038 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.503943    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-credential-socket\") pod \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") "
Oct 31 00:54:55.504179 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.504119    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istiod-ca-cert\") pod \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") "
Oct 31 00:54:55.504285 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.504234    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-workload-socket\") pod \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") "
Oct 31 00:54:55.504460 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.504335    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-podinfo\") pod \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\" (UID: \"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770\") "
Oct 31 00:54:55.508963 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.508931    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-envoy" pod "c5c6a0bf-623b-41e0-9c60-4a3fa55e8770" (UID: "c5c6a0bf-623b-41e0-9c60-4a3fa55e8770"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:54:55.509107 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.509085    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-credential-socket" pod "c5c6a0bf-623b-41e0-9c60-4a3fa55e8770" (UID: "c5c6a0bf-623b-41e0-9c60-4a3fa55e8770"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:54:55.509732 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.509605    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-workload-socket" pod "c5c6a0bf-623b-41e0-9c60-4a3fa55e8770" (UID: "c5c6a0bf-623b-41e0-9c60-4a3fa55e8770"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:54:55.510643 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.510457    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istiod-ca-cert" pod "c5c6a0bf-623b-41e0-9c60-4a3fa55e8770" (UID: "c5c6a0bf-623b-41e0-9c60-4a3fa55e8770"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:54:55.511146 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.510930    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-token" pod "c5c6a0bf-623b-41e0-9c60-4a3fa55e8770" (UID: "c5c6a0bf-623b-41e0-9c60-4a3fa55e8770"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:54:55.511795 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.511753    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-podinfo" pod "c5c6a0bf-623b-41e0-9c60-4a3fa55e8770" (UID: "c5c6a0bf-623b-41e0-9c60-4a3fa55e8770"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:54:55.531925 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.531898    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:55.532111 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.532095    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:55.532194 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.532181    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:55.532289 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.532275    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:55.532418 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.532400    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:55.532499 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.532487    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:55.532615 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.532568    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:55.532756 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.532740    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:55.539071 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.539028    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-kube-api-access-lrswv" pod "c5c6a0bf-623b-41e0-9c60-4a3fa55e8770" (UID: "c5c6a0bf-623b-41e0-9c60-4a3fa55e8770"). InnerVolumeSpecName "kube-api-access-lrswv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:54:55.634801 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:55.634764    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrswv\" (UniqueName: \"kubernetes.io/projected/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770-kube-api-access-lrswv\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:56.025493 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.025406    4327 generic.go:356] "Generic (PLEG): container finished" podID="1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c" containerID="c5165caf7853cd703ac494fc85f1a4f5dd18a0cda7d66b1e4d497148e5135681" exitCode=0
Oct 31 00:54:56.025979 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.025936    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9" event={"ID":"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c","Type":"ContainerDied","Data":"c5165caf7853cd703ac494fc85f1a4f5dd18a0cda7d66b1e4d497148e5135681"}
Oct 31 00:54:56.026230 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.026181    4327 scope.go:122] "RemoveContainer" containerID="c48de8d5b4670edd4e87a5b6da7325a7b65e59159f2771c8f9223b620b93c315"
Oct 31 00:54:56.035175 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.035142    4327 generic.go:356] "Generic (PLEG): container finished" podID="c5c6a0bf-623b-41e0-9c60-4a3fa55e8770" containerID="9a185e9c58176b62d3a4e9195aa9e1607687144d3bbc1d0ba299ace77be5fb96" exitCode=0
Oct 31 00:54:56.035458 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.035438    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7" event={"ID":"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770","Type":"ContainerDied","Data":"9a185e9c58176b62d3a4e9195aa9e1607687144d3bbc1d0ba299ace77be5fb96"}
Oct 31 00:54:56.035660 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.035621    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7" event={"ID":"c5c6a0bf-623b-41e0-9c60-4a3fa55e8770","Type":"ContainerDied","Data":"20e2467129d098feeb82df5403873abf9922ea47faf4d70a77de3db33ab59df8"}
Oct 31 00:54:56.036406 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.036379    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"
Oct 31 00:54:56.053075 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.053041    4327 generic.go:356] "Generic (PLEG): container finished" podID="242be892-3b92-44cb-a8ef-17f8ac581b4b" containerID="352a138edc15db7342152a37d38e4c6ff57b41b971fe361b2276fa3ee8960229" exitCode=0
Oct 31 00:54:56.082740 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.054217    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk" event={"ID":"242be892-3b92-44cb-a8ef-17f8ac581b4b","Type":"ContainerDied","Data":"352a138edc15db7342152a37d38e4c6ff57b41b971fe361b2276fa3ee8960229"}
Oct 31 00:54:56.086378 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.086350    4327 scope.go:122] "RemoveContainer" containerID="352a138edc15db7342152a37d38e4c6ff57b41b971fe361b2276fa3ee8960229"
Oct 31 00:54:56.091384 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.091335    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:54:56.091525 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.091449    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:54:56.091525 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.091501    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:54:56.112971 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.112947    4327 scope.go:122] "RemoveContainer" containerID="9a185e9c58176b62d3a4e9195aa9e1607687144d3bbc1d0ba299ace77be5fb96"
Oct 31 00:54:56.125946 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.125855    4327 scope.go:122] "RemoveContainer" containerID="9a185e9c58176b62d3a4e9195aa9e1607687144d3bbc1d0ba299ace77be5fb96"
Oct 31 00:54:56.126372 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:54:56.126348    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a185e9c58176b62d3a4e9195aa9e1607687144d3bbc1d0ba299ace77be5fb96\": container with ID starting with 9a185e9c58176b62d3a4e9195aa9e1607687144d3bbc1d0ba299ace77be5fb96 not found: ID does not exist" containerID="9a185e9c58176b62d3a4e9195aa9e1607687144d3bbc1d0ba299ace77be5fb96"
Oct 31 00:54:56.126475 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.126383    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a185e9c58176b62d3a4e9195aa9e1607687144d3bbc1d0ba299ace77be5fb96"} err="failed to get container status \"9a185e9c58176b62d3a4e9195aa9e1607687144d3bbc1d0ba299ace77be5fb96\": rpc error: code = NotFound desc = could not find container \"9a185e9c58176b62d3a4e9195aa9e1607687144d3bbc1d0ba299ace77be5fb96\": container with ID starting with 9a185e9c58176b62d3a4e9195aa9e1607687144d3bbc1d0ba299ace77be5fb96 not found: ID does not exist"
Oct 31 00:54:56.179371 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.179338    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"]
Oct 31 00:54:56.185210 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.185180    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t9ds7"]
Oct 31 00:54:56.299027 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.298965    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:54:56.466988 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.466943    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-workload-socket\") pod \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") "
Oct 31 00:54:56.467505 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.467038    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-podinfo\") pod \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") "
Oct 31 00:54:56.467505 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.467163    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-credential-socket\") pod \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") "
Oct 31 00:54:56.467505 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.467249    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-data\") pod \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") "
Oct 31 00:54:56.467505 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.467282    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-envoy\") pod \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") "
Oct 31 00:54:56.467505 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.467340    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istiod-ca-cert\") pod \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") "
Oct 31 00:54:56.467505 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.467404    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-kube-api-access-kzxxl\" (UniqueName: \"kubernetes.io/projected/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-kube-api-access-kzxxl\") pod \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") "
Oct 31 00:54:56.467505 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.467467    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-token\" (UniqueName: \"kubernetes.io/projected/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-token\") pod \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") "
Oct 31 00:54:56.467505 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.467504    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-workload-certs\") pod \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\" (UID: \"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c\") "
Oct 31 00:54:56.468620 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.468584    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-workload-certs" pod "1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c" (UID: "1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:54:56.469042 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.468850    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-workload-socket" pod "1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c" (UID: "1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:54:56.470951 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.469843    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istiod-ca-cert" pod "1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c" (UID: "1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:54:56.470951 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.470531    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-data" pod "1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c" (UID: "1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:54:56.470951 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.470766    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-credential-socket" pod "1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c" (UID: "1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:54:56.471499 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.471472    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-podinfo" pod "1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c" (UID: "1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:54:56.472170 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.472140    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-kube-api-access-kzxxl" pod "1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c" (UID: "1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c"). InnerVolumeSpecName "kube-api-access-kzxxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:54:56.473361 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.473335    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-token" pod "1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c" (UID: "1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:54:56.484201 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.484170    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-envoy" pod "1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c" (UID: "1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:54:56.569526 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.569438    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:56.569526 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.569472    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:56.569526 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.569481    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:56.569526 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.569490    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:56.569526 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.569498    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:56.569526 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.569507    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:56.569526 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.569515    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzxxl\" (UniqueName: \"kubernetes.io/projected/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-kube-api-access-kzxxl\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:56.569526 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.569523    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:56.569526 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.569531    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:54:56.724948 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.700482    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"]
Oct 31 00:54:56.753294 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.753254    4327 cpu_manager.go:414] "containerMap: removing container" podUID="c5c6a0bf-623b-41e0-9c60-4a3fa55e8770" containerName="istio-proxy"
Oct 31 00:54:56.753597 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.753573    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c6a0bf-623b-41e0-9c60-4a3fa55e8770" containerName="istio-proxy"
Oct 31 00:54:56.753769 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.753754    4327 cpu_manager.go:414] "containerMap: removing container" podUID="1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c" containerName="istio-proxy"
Oct 31 00:54:56.753868 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.753857    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c" containerName="istio-proxy"
Oct 31 00:54:56.753987 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.753977    4327 cpu_manager.go:414] "containerMap: removing container" podUID="1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c" containerName="istio-proxy"
Oct 31 00:54:56.754052 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.754043    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c" containerName="istio-proxy"
Oct 31 00:54:56.755079 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.755059    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c" containerName="istio-proxy"
Oct 31 00:54:56.755238 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.755226    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c6a0bf-623b-41e0-9c60-4a3fa55e8770" containerName="istio-proxy"
Oct 31 00:54:56.799703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.792016    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:56.870755 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.869976    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9de7f5-openshift-default-dockercfg-h65hq\""
Oct 31 00:54:56.929304 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.928328    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"]
Oct 31 00:54:56.955593 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.955556    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b046f39f-ed7f-414e-aedf-be830e648883-istiod-ca-cert\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:56.955884 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.955859    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zxxs\" (UniqueName: \"kubernetes.io/projected/b046f39f-ed7f-414e-aedf-be830e648883-kube-api-access-5zxxs\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:56.956000 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.955905    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b046f39f-ed7f-414e-aedf-be830e648883-istio-podinfo\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:56.956148 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.956119    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-istio-envoy\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:56.956226 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.956193    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-istio-data\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:56.956496 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.956477    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b046f39f-ed7f-414e-aedf-be830e648883-istio-token\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:56.956622 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.956607    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-workload-certs\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:56.956708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.956653    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-credential-socket\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:56.956708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:56.956698    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-workload-socket\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:57.067704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.065312    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b046f39f-ed7f-414e-aedf-be830e648883-istiod-ca-cert\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:57.067704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.065422    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zxxs\" (UniqueName: \"kubernetes.io/projected/b046f39f-ed7f-414e-aedf-be830e648883-kube-api-access-5zxxs\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:57.067704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.065461    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b046f39f-ed7f-414e-aedf-be830e648883-istio-podinfo\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:57.067704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.065581    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-istio-envoy\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:57.067704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.065643    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-istio-data\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:57.067704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.065839    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b046f39f-ed7f-414e-aedf-be830e648883-istio-token\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:57.067704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.065949    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-workload-certs\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:57.067704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.065982    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-credential-socket\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:57.067704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.066009    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-workload-socket\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:57.067704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.066596    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-workload-socket\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:57.084247 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.083840    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b046f39f-ed7f-414e-aedf-be830e648883-istiod-ca-cert\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:57.084247 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.084200    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-credential-socket\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:57.084637 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.084617    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-workload-certs\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:57.085013 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.084976    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-istio-data\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:57.085225 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.085210    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-istio-envoy\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:57.097791 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.095278    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b046f39f-ed7f-414e-aedf-be830e648883-istio-podinfo\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:57.105430 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.105367    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zxxs\" (UniqueName: \"kubernetes.io/projected/b046f39f-ed7f-414e-aedf-be830e648883-kube-api-access-5zxxs\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:57.108539 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.108508    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk" event={"ID":"242be892-3b92-44cb-a8ef-17f8ac581b4b","Type":"ContainerStarted","Data":"fec6de9c040ea38c58718a3e1ed0cd8ae305e29eca555895eeb1d7db0df3c80e"}
Oct 31 00:54:57.110191 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.110170    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b046f39f-ed7f-414e-aedf-be830e648883-istio-token\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:57.111330 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.111312    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"
Oct 31 00:54:57.111419 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.111373    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9" event={"ID":"1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c","Type":"ContainerDied","Data":"70e5f1191b543710e4625b79b50ca3526286954718060fa334cb75d71c1a6caa"}
Oct 31 00:54:57.111467 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.111429    4327 scope.go:122] "RemoveContainer" containerID="c5165caf7853cd703ac494fc85f1a4f5dd18a0cda7d66b1e4d497148e5135681"
Oct 31 00:54:57.212151 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.212030    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:57.407770 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.407455    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"]
Oct 31 00:54:57.416128 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.416068    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-vjpz9"]
Oct 31 00:54:57.748703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.748661    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"]
Oct 31 00:54:57.749490 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:54:57.749430    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb046f39f_ed7f_414e_aedf_be830e648883.slice/crio-cc70f6dbd31a41ba36c92477b6f13c857f5a3bcebcc3fcb8dc87d31ba9bc53bc WatchSource:0}: Error finding container cc70f6dbd31a41ba36c92477b6f13c857f5a3bcebcc3fcb8dc87d31ba9bc53bc: Status 404 returned error can't find the container with id cc70f6dbd31a41ba36c92477b6f13c857f5a3bcebcc3fcb8dc87d31ba9bc53bc
Oct 31 00:54:57.752575 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.752534    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:54:57.752703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.752636    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:54:57.752777 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.752709    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:54:57.788040 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.788000    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8"]
Oct 31 00:54:57.789034 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.789009    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c" containerName="istio-proxy"
Oct 31 00:54:57.794411 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.794386    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8"
Oct 31 00:54:57.811062 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.811009    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-01a117-dockercfg-bbblj\""
Oct 31 00:54:57.811749 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.811730    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-01a117\""
Oct 31 00:54:57.814351 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.814324    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57e3b4e5-4f46-4c99-a260-3beb556431c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8\" (UID: \"57e3b4e5-4f46-4c99-a260-3beb556431c7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8"
Oct 31 00:54:57.814484 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.814469    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtjq8\" (UniqueName: \"kubernetes.io/projected/57e3b4e5-4f46-4c99-a260-3beb556431c7-kube-api-access-xtjq8\") pod \"payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8\" (UID: \"57e3b4e5-4f46-4c99-a260-3beb556431c7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8"
Oct 31 00:54:57.816415 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.816392    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8"]
Oct 31 00:54:57.928756 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.922952    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57e3b4e5-4f46-4c99-a260-3beb556431c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8\" (UID: \"57e3b4e5-4f46-4c99-a260-3beb556431c7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8"
Oct 31 00:54:57.928756 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.923068    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtjq8\" (UniqueName: \"kubernetes.io/projected/57e3b4e5-4f46-4c99-a260-3beb556431c7-kube-api-access-xtjq8\") pod \"payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8\" (UID: \"57e3b4e5-4f46-4c99-a260-3beb556431c7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8"
Oct 31 00:54:57.936704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.934253    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57e3b4e5-4f46-4c99-a260-3beb556431c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8\" (UID: \"57e3b4e5-4f46-4c99-a260-3beb556431c7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8"
Oct 31 00:54:58.006390 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.999877    4327 patch_prober.go:29] interesting pod/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 00:54:58.006390 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:57.999939    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk" podUID="242be892-3b92-44cb-a8ef-17f8ac581b4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 00:54:58.271808 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:58.263738    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtjq8\" (UniqueName: \"kubernetes.io/projected/57e3b4e5-4f46-4c99-a260-3beb556431c7-kube-api-access-xtjq8\") pod \"payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8\" (UID: \"57e3b4e5-4f46-4c99-a260-3beb556431c7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8"
Oct 31 00:54:58.311562 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:58.311505    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c" path="/var/lib/kubelet/pods/1f9d0a6b-3a17-4869-8dcb-3152a72a1a7c/volumes"
Oct 31 00:54:58.313199 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:58.313172    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5c6a0bf-623b-41e0-9c60-4a3fa55e8770" path="/var/lib/kubelet/pods/c5c6a0bf-623b-41e0-9c60-4a3fa55e8770/volumes"
Oct 31 00:54:58.313822 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:58.313799    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:54:58.341745 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:58.341661    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf"]
Oct 31 00:54:58.370044 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:58.370010    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf"]
Oct 31 00:54:58.370287 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:58.370115    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf"
Oct 31 00:54:58.376461 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:58.376419    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt" event={"ID":"b046f39f-ed7f-414e-aedf-be830e648883","Type":"ContainerStarted","Data":"cc70f6dbd31a41ba36c92477b6f13c857f5a3bcebcc3fcb8dc87d31ba9bc53bc"}
Oct 31 00:54:58.414344 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:58.414309    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8"
Oct 31 00:54:58.471566 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:58.471490    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt" podStartSLOduration=2.47147056 podStartE2EDuration="2.47147056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:58.463941696 +0000 UTC m=+2158.685977237" watchObservedRunningTime="2026-10-31 00:54:58.47147056 +0000 UTC m=+2158.693506115"
Oct 31 00:54:58.471758 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:58.471565    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf355564-5ea9-4149-9540-d01964674903-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf\" (UID: \"cf355564-5ea9-4149-9540-d01964674903\") " pod="openshift-ingress/payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf"
Oct 31 00:54:58.473726 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:58.472142    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhn8l\" (UniqueName: \"kubernetes.io/projected/cf355564-5ea9-4149-9540-d01964674903-kube-api-access-xhn8l\") pod \"payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf\" (UID: \"cf355564-5ea9-4149-9540-d01964674903\") " pod="openshift-ingress/payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf"
Oct 31 00:54:58.573286 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:58.573248    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhn8l\" (UniqueName: \"kubernetes.io/projected/cf355564-5ea9-4149-9540-d01964674903-kube-api-access-xhn8l\") pod \"payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf\" (UID: \"cf355564-5ea9-4149-9540-d01964674903\") " pod="openshift-ingress/payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf"
Oct 31 00:54:58.573472 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:58.573337    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf355564-5ea9-4149-9540-d01964674903-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf\" (UID: \"cf355564-5ea9-4149-9540-d01964674903\") " pod="openshift-ingress/payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf"
Oct 31 00:54:58.600728 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:58.584077    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf355564-5ea9-4149-9540-d01964674903-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf\" (UID: \"cf355564-5ea9-4149-9540-d01964674903\") " pod="openshift-ingress/payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf"
Oct 31 00:54:58.731643 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:58.731565    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhn8l\" (UniqueName: \"kubernetes.io/projected/cf355564-5ea9-4149-9540-d01964674903-kube-api-access-xhn8l\") pod \"payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf\" (UID: \"cf355564-5ea9-4149-9540-d01964674903\") " pod="openshift-ingress/payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf"
Oct 31 00:54:58.823126 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:58.823034    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8"]
Oct 31 00:54:58.922481 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:54:58.911882    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e3b4e5_4f46_4c99_a260_3beb556431c7.slice/crio-edd1d6492be78468f91a946c9bd9168633fe54906f66b4ef2146efcadf7358bc WatchSource:0}: Error finding container edd1d6492be78468f91a946c9bd9168633fe54906f66b4ef2146efcadf7358bc: Status 404 returned error can't find the container with id edd1d6492be78468f91a946c9bd9168633fe54906f66b4ef2146efcadf7358bc
Oct 31 00:54:58.942916 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:58.942879    4327 patch_prober.go:29] interesting pod/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 00:54:58.943205 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:58.943176    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk" podUID="242be892-3b92-44cb-a8ef-17f8ac581b4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 00:54:58.990957 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:58.990929    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf"
Oct 31 00:54:59.216361 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:59.216322    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:59.240071 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:59.240038    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:59.375025 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:59.374989    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf"]
Oct 31 00:54:59.384353 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:54:59.384322    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf355564_5ea9_4149_9540_d01964674903.slice/crio-00e41492e2ebad59685cb18e313a0d92ff805a708ead6bf3b00f727d981368ff WatchSource:0}: Error finding container 00e41492e2ebad59685cb18e313a0d92ff805a708ead6bf3b00f727d981368ff: Status 404 returned error can't find the container with id 00e41492e2ebad59685cb18e313a0d92ff805a708ead6bf3b00f727d981368ff
Oct 31 00:54:59.391170 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:59.391123    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt" event={"ID":"b046f39f-ed7f-414e-aedf-be830e648883","Type":"ContainerStarted","Data":"5984429b31752f91e808ec4c27cdac411d637af501d998cac60b16cb03a39e36"}
Oct 31 00:54:59.392534 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:59.392393    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:59.394322 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:59.394301    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8" event={"ID":"57e3b4e5-4f46-4c99-a260-3beb556431c7","Type":"ContainerStarted","Data":"40a46409a803826b1ff0f52d090826c434c2c15eeb9b89e6ac58a7c17dd8712c"}
Oct 31 00:54:59.394482 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:59.394467    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8"
Oct 31 00:54:59.394615 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:59.394599    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8" event={"ID":"57e3b4e5-4f46-4c99-a260-3beb556431c7","Type":"ContainerStarted","Data":"edd1d6492be78468f91a946c9bd9168633fe54906f66b4ef2146efcadf7358bc"}
Oct 31 00:54:59.397422 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:59.397399    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:54:59.450052 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:59.449990    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8" podStartSLOduration=2.449973106 podStartE2EDuration="2.449973106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:59.449166607 +0000 UTC m=+2159.671202149" watchObservedRunningTime="2026-10-31 00:54:59.449973106 +0000 UTC m=+2159.672008649"
Oct 31 00:54:59.946137 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:59.946099    4327 patch_prober.go:29] interesting pod/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 00:54:59.946741 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:54:59.946707    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk" podUID="242be892-3b92-44cb-a8ef-17f8ac581b4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 00:55:00.404520 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:00.404465    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf" event={"ID":"cf355564-5ea9-4149-9540-d01964674903","Type":"ContainerStarted","Data":"f7710cac8afed708d11351d308206a1467d68be7db6fb7734881fc91e94629f6"}
Oct 31 00:55:00.404751 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:00.404534    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf" event={"ID":"cf355564-5ea9-4149-9540-d01964674903","Type":"ContainerStarted","Data":"00e41492e2ebad59685cb18e313a0d92ff805a708ead6bf3b00f727d981368ff"}
Oct 31 00:55:00.404866 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:00.404845    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf"
Oct 31 00:55:00.465940 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:00.465878    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf" podStartSLOduration=3.465863264 podStartE2EDuration="3.465863264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:00.458709527 +0000 UTC m=+2160.680745070" watchObservedRunningTime="2026-10-31 00:55:00.465863264 +0000 UTC m=+2160.687898806"
Oct 31 00:55:00.943254 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:00.943202    4327 patch_prober.go:29] interesting pod/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 00:55:00.943424 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:00.943277    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk" podUID="242be892-3b92-44cb-a8ef-17f8ac581b4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 00:55:01.948705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:01.944786    4327 patch_prober.go:29] interesting pod/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 00:55:01.948705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:01.944853    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk" podUID="242be892-3b92-44cb-a8ef-17f8ac581b4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 00:55:02.949383 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:02.949270    4327 patch_prober.go:29] interesting pod/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 00:55:02.949383 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:02.949324    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk" podUID="242be892-3b92-44cb-a8ef-17f8ac581b4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 00:55:03.039073 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:03.039037    4327 scope.go:122] "RemoveContainer" containerID="eee15371f50606cf697279bd22df73dba21b52dc03af3c127bbbf9c958aedc75"
Oct 31 00:55:03.164742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:03.163596    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm"]
Oct 31 00:55:03.171547 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:03.171518    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm"
Oct 31 00:55:03.190460 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:03.187612    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9de7f5\""
Oct 31 00:55:03.192908 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:03.192877    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm"]
Oct 31 00:55:03.208793 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:03.207957    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9de7f5-dockercfg-66gbr\""
Oct 31 00:55:03.284713 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:03.282762    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk7m5\" (UniqueName: \"kubernetes.io/projected/67a2779b-8ad1-40f8-b30c-d55209a1421d-kube-api-access-gk7m5\") pod \"payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm\" (UID: \"67a2779b-8ad1-40f8-b30c-d55209a1421d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm"
Oct 31 00:55:03.284713 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:03.283258    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a2779b-8ad1-40f8-b30c-d55209a1421d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm\" (UID: \"67a2779b-8ad1-40f8-b30c-d55209a1421d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm"
Oct 31 00:55:03.487050 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:03.407535    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk7m5\" (UniqueName: \"kubernetes.io/projected/67a2779b-8ad1-40f8-b30c-d55209a1421d-kube-api-access-gk7m5\") pod \"payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm\" (UID: \"67a2779b-8ad1-40f8-b30c-d55209a1421d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm"
Oct 31 00:55:03.487050 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:03.472006    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a2779b-8ad1-40f8-b30c-d55209a1421d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm\" (UID: \"67a2779b-8ad1-40f8-b30c-d55209a1421d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm"
Oct 31 00:55:03.487050 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:03.473466    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a2779b-8ad1-40f8-b30c-d55209a1421d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm\" (UID: \"67a2779b-8ad1-40f8-b30c-d55209a1421d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm"
Oct 31 00:55:03.635882 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:03.635843    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk7m5\" (UniqueName: \"kubernetes.io/projected/67a2779b-8ad1-40f8-b30c-d55209a1421d-kube-api-access-gk7m5\") pod \"payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm\" (UID: \"67a2779b-8ad1-40f8-b30c-d55209a1421d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm"
Oct 31 00:55:03.800417 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:03.799593    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm"
Oct 31 00:55:03.955178 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:03.955146    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:55:03.956264 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:03.956237    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:55:03.957824 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:03.957803    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 00:55:04.138177 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:04.138146    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm"]
Oct 31 00:55:04.138907 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:55:04.138771    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a2779b_8ad1_40f8_b30c_d55209a1421d.slice/crio-0b1fa31d4c739bfe685f8bb981adbe2cdd1ab0953eb77105030c83d427265fd4 WatchSource:0}: Error finding container 0b1fa31d4c739bfe685f8bb981adbe2cdd1ab0953eb77105030c83d427265fd4: Status 404 returned error can't find the container with id 0b1fa31d4c739bfe685f8bb981adbe2cdd1ab0953eb77105030c83d427265fd4
Oct 31 00:55:04.651775 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:04.651719    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm" event={"ID":"67a2779b-8ad1-40f8-b30c-d55209a1421d","Type":"ContainerStarted","Data":"c8368b84b77bdaca669587725e9b5760dc3a07d834ff8fd5f100d203054e23fc"}
Oct 31 00:55:04.651975 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:04.651796    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm" event={"ID":"67a2779b-8ad1-40f8-b30c-d55209a1421d","Type":"ContainerStarted","Data":"0b1fa31d4c739bfe685f8bb981adbe2cdd1ab0953eb77105030c83d427265fd4"}
Oct 31 00:55:04.652041 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:04.652028    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm"
Oct 31 00:55:04.697543 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:04.697477    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm" podStartSLOduration=1.697456031 podStartE2EDuration="1.697456031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:04.693825456 +0000 UTC m=+2164.915860999" watchObservedRunningTime="2026-10-31 00:55:04.697456031 +0000 UTC m=+2164.919491577"
Oct 31 00:55:08.209794 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:08.209757    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79"]
Oct 31 00:55:08.214919 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:08.214896    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79"
Oct 31 00:55:08.251556 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:08.251515    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-72120c\""
Oct 31 00:55:08.252293 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:08.252255    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79"]
Oct 31 00:55:08.274704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:08.261015    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dfpn\" (UniqueName: \"kubernetes.io/projected/0f28a16d-689f-4baf-96ac-c2c9e7908888-kube-api-access-2dfpn\") pod \"payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79\" (UID: \"0f28a16d-689f-4baf-96ac-c2c9e7908888\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79"
Oct 31 00:55:08.275160 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:08.275126    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f28a16d-689f-4baf-96ac-c2c9e7908888-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79\" (UID: \"0f28a16d-689f-4baf-96ac-c2c9e7908888\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79"
Oct 31 00:55:08.339470 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:08.325542    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-72120c-dockercfg-4xwlj\""
Oct 31 00:55:08.413704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:08.389148    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dfpn\" (UniqueName: \"kubernetes.io/projected/0f28a16d-689f-4baf-96ac-c2c9e7908888-kube-api-access-2dfpn\") pod \"payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79\" (UID: \"0f28a16d-689f-4baf-96ac-c2c9e7908888\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79"
Oct 31 00:55:08.432711 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:08.420959    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f28a16d-689f-4baf-96ac-c2c9e7908888-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79\" (UID: \"0f28a16d-689f-4baf-96ac-c2c9e7908888\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79"
Oct 31 00:55:08.432711 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:08.422067    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f28a16d-689f-4baf-96ac-c2c9e7908888-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79\" (UID: \"0f28a16d-689f-4baf-96ac-c2c9e7908888\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79"
Oct 31 00:55:08.623707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:08.620867    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd"]
Oct 31 00:55:08.632429 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:08.629496    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd"
Oct 31 00:55:08.632429 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:08.632286    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd"]
Oct 31 00:55:08.634774 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:08.634746    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dfpn\" (UniqueName: \"kubernetes.io/projected/0f28a16d-689f-4baf-96ac-c2c9e7908888-kube-api-access-2dfpn\") pod \"payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79\" (UID: \"0f28a16d-689f-4baf-96ac-c2c9e7908888\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79"
Oct 31 00:55:08.716268 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:08.716217    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0715eb-599c-4663-874c-b350f7a2910d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd\" (UID: \"9b0715eb-599c-4663-874c-b350f7a2910d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd"
Oct 31 00:55:08.716454 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:08.716377    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-587rg\" (UniqueName: \"kubernetes.io/projected/9b0715eb-599c-4663-874c-b350f7a2910d-kube-api-access-587rg\") pod \"payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd\" (UID: \"9b0715eb-599c-4663-874c-b350f7a2910d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd"
Oct 31 00:55:08.818350 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:08.818307    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0715eb-599c-4663-874c-b350f7a2910d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd\" (UID: \"9b0715eb-599c-4663-874c-b350f7a2910d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd"
Oct 31 00:55:08.818542 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:08.818428    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-587rg\" (UniqueName: \"kubernetes.io/projected/9b0715eb-599c-4663-874c-b350f7a2910d-kube-api-access-587rg\") pod \"payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd\" (UID: \"9b0715eb-599c-4663-874c-b350f7a2910d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd"
Oct 31 00:55:08.819134 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:08.819108    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0715eb-599c-4663-874c-b350f7a2910d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd\" (UID: \"9b0715eb-599c-4663-874c-b350f7a2910d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd"
Oct 31 00:55:08.842247 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:08.842216    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-587rg\" (UniqueName: \"kubernetes.io/projected/9b0715eb-599c-4663-874c-b350f7a2910d-kube-api-access-587rg\") pod \"payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd\" (UID: \"9b0715eb-599c-4663-874c-b350f7a2910d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd"
Oct 31 00:55:08.853108 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:08.853077    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79"
Oct 31 00:55:08.965196 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:08.964866    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd"
Oct 31 00:55:09.163319 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:09.163264    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79"]
Oct 31 00:55:09.163883 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:55:09.163853    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f28a16d_689f_4baf_96ac_c2c9e7908888.slice/crio-0e0abcfaa66d3910d089da45871f7fbc0d628f4a2121e1bc7b08d074a80edb35 WatchSource:0}: Error finding container 0e0abcfaa66d3910d089da45871f7fbc0d628f4a2121e1bc7b08d074a80edb35: Status 404 returned error can't find the container with id 0e0abcfaa66d3910d089da45871f7fbc0d628f4a2121e1bc7b08d074a80edb35
Oct 31 00:55:09.287922 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:55:09.287888    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0715eb_599c_4663_874c_b350f7a2910d.slice/crio-fd328b4298b0b41dcb0fbf16417c62f60d0e8f033e753e8692d3e5980f4ddd93 WatchSource:0}: Error finding container fd328b4298b0b41dcb0fbf16417c62f60d0e8f033e753e8692d3e5980f4ddd93: Status 404 returned error can't find the container with id fd328b4298b0b41dcb0fbf16417c62f60d0e8f033e753e8692d3e5980f4ddd93
Oct 31 00:55:09.288304 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:09.288032    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd"]
Oct 31 00:55:09.693987 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:09.693947    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79" event={"ID":"0f28a16d-689f-4baf-96ac-c2c9e7908888","Type":"ContainerStarted","Data":"5594c9583f5073f1d392ece714a8d3fe3ee0cafac91459c9942fc30e3d35d4ba"}
Oct 31 00:55:09.694165 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:09.694013    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79" event={"ID":"0f28a16d-689f-4baf-96ac-c2c9e7908888","Type":"ContainerStarted","Data":"0e0abcfaa66d3910d089da45871f7fbc0d628f4a2121e1bc7b08d074a80edb35"}
Oct 31 00:55:09.694165 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:09.694043    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79"
Oct 31 00:55:09.695867 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:09.695838    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd" event={"ID":"9b0715eb-599c-4663-874c-b350f7a2910d","Type":"ContainerStarted","Data":"add5fa3e5de12e3c452c9c4ed39c39ea47f54b10873a2db002995a724c2428c8"}
Oct 31 00:55:09.695995 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:09.695893    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd" event={"ID":"9b0715eb-599c-4663-874c-b350f7a2910d","Type":"ContainerStarted","Data":"fd328b4298b0b41dcb0fbf16417c62f60d0e8f033e753e8692d3e5980f4ddd93"}
Oct 31 00:55:09.695995 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:09.695924    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd"
Oct 31 00:55:09.740060 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:09.739994    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79" podStartSLOduration=1.73997582 podStartE2EDuration="1.73997582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:09.735785316 +0000 UTC m=+2169.957820856" watchObservedRunningTime="2026-10-31 00:55:09.73997582 +0000 UTC m=+2169.962011364"
Oct 31 00:55:09.801568 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:09.801512    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd" podStartSLOduration=1.801497181 podStartE2EDuration="1.801497181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:09.797033433 +0000 UTC m=+2170.019068976" watchObservedRunningTime="2026-10-31 00:55:09.801497181 +0000 UTC m=+2170.023532724"
Oct 31 00:55:10.407850 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:10.407824    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8"
Oct 31 00:55:11.484656 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:11.484627    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf"
Oct 31 00:55:15.296128 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:55:15.296084    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:55:15.662158 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:15.662128    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm"
Oct 31 00:55:20.707159 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:20.707124    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79"
Oct 31 00:55:20.707655 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:20.707208    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd"
Oct 31 00:55:24.751485 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:24.751449    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl"]
Oct 31 00:55:24.752064 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:24.751666    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl" podUID="b1783fef-e971-4cce-861a-37b8c232d0b3" containerName="maas-api" containerID="cri-o://a2775b9ce46bd8bcb696c627a3cde8db6dc94a2c872814b1f16908dee43f43ed" gracePeriod=30
Oct 31 00:55:25.359708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:25.353178    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl"
Oct 31 00:55:25.448511 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:25.448468    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1783fef-e971-4cce-861a-37b8c232d0b3-kube-api-access-rn695\" (UniqueName: \"kubernetes.io/projected/b1783fef-e971-4cce-861a-37b8c232d0b3-kube-api-access-rn695\") pod \"b1783fef-e971-4cce-861a-37b8c232d0b3\" (UID: \"b1783fef-e971-4cce-861a-37b8c232d0b3\") "
Oct 31 00:55:25.449370 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:25.449345    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1783fef-e971-4cce-861a-37b8c232d0b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1783fef-e971-4cce-861a-37b8c232d0b3-openshift-service-ca\") pod \"b1783fef-e971-4cce-861a-37b8c232d0b3\" (UID: \"b1783fef-e971-4cce-861a-37b8c232d0b3\") "
Oct 31 00:55:25.449662 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:25.449643    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1783fef-e971-4cce-861a-37b8c232d0b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1783fef-e971-4cce-861a-37b8c232d0b3-maas-api-tls\") pod \"b1783fef-e971-4cce-861a-37b8c232d0b3\" (UID: \"b1783fef-e971-4cce-861a-37b8c232d0b3\") "
Oct 31 00:55:25.449841 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:25.449822    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1783fef-e971-4cce-861a-37b8c232d0b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1783fef-e971-4cce-861a-37b8c232d0b3-metrics-certs\") pod \"b1783fef-e971-4cce-861a-37b8c232d0b3\" (UID: \"b1783fef-e971-4cce-861a-37b8c232d0b3\") "
Oct 31 00:55:25.452106 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:25.451799    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1783fef-e971-4cce-861a-37b8c232d0b3-openshift-service-ca" pod "b1783fef-e971-4cce-861a-37b8c232d0b3" (UID: "b1783fef-e971-4cce-861a-37b8c232d0b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:55:25.458347 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:25.458319    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1783fef-e971-4cce-861a-37b8c232d0b3-metrics-certs" pod "b1783fef-e971-4cce-861a-37b8c232d0b3" (UID: "b1783fef-e971-4cce-861a-37b8c232d0b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:55:25.458662 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:25.458632    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1783fef-e971-4cce-861a-37b8c232d0b3-kube-api-access-rn695" pod "b1783fef-e971-4cce-861a-37b8c232d0b3" (UID: "b1783fef-e971-4cce-861a-37b8c232d0b3"). InnerVolumeSpecName "kube-api-access-rn695". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:55:25.506770 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:25.496359    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1783fef-e971-4cce-861a-37b8c232d0b3-maas-api-tls" pod "b1783fef-e971-4cce-861a-37b8c232d0b3" (UID: "b1783fef-e971-4cce-861a-37b8c232d0b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:55:25.567503 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:25.559430    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1783fef-e971-4cce-861a-37b8c232d0b3-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:55:25.567503 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:25.559462    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1783fef-e971-4cce-861a-37b8c232d0b3-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:55:25.567503 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:25.559477    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1783fef-e971-4cce-861a-37b8c232d0b3-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:55:25.567503 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:25.559490    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn695\" (UniqueName: \"kubernetes.io/projected/b1783fef-e971-4cce-861a-37b8c232d0b3-kube-api-access-rn695\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:55:25.861700 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:25.860287    4327 generic.go:356] "Generic (PLEG): container finished" podID="b1783fef-e971-4cce-861a-37b8c232d0b3" containerID="a2775b9ce46bd8bcb696c627a3cde8db6dc94a2c872814b1f16908dee43f43ed" exitCode=0
Oct 31 00:55:25.861700 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:25.860336    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl" event={"ID":"b1783fef-e971-4cce-861a-37b8c232d0b3","Type":"ContainerDied","Data":"a2775b9ce46bd8bcb696c627a3cde8db6dc94a2c872814b1f16908dee43f43ed"}
Oct 31 00:55:25.861700 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:25.860361    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl"
Oct 31 00:55:25.861700 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:25.860427    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl" event={"ID":"b1783fef-e971-4cce-861a-37b8c232d0b3","Type":"ContainerDied","Data":"5815b9af6225c4b1fdf2b847596d3b23ded9ca638f09ea3930ec7fbb4cc4c39b"}
Oct 31 00:55:25.861700 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:25.860472    4327 scope.go:122] "RemoveContainer" containerID="a2775b9ce46bd8bcb696c627a3cde8db6dc94a2c872814b1f16908dee43f43ed"
Oct 31 00:55:25.874143 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:25.874123    4327 scope.go:122] "RemoveContainer" containerID="a2775b9ce46bd8bcb696c627a3cde8db6dc94a2c872814b1f16908dee43f43ed"
Oct 31 00:55:25.874502 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:55:25.874476    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2775b9ce46bd8bcb696c627a3cde8db6dc94a2c872814b1f16908dee43f43ed\": container with ID starting with a2775b9ce46bd8bcb696c627a3cde8db6dc94a2c872814b1f16908dee43f43ed not found: ID does not exist" containerID="a2775b9ce46bd8bcb696c627a3cde8db6dc94a2c872814b1f16908dee43f43ed"
Oct 31 00:55:25.874614 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:25.874509    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2775b9ce46bd8bcb696c627a3cde8db6dc94a2c872814b1f16908dee43f43ed"} err="failed to get container status \"a2775b9ce46bd8bcb696c627a3cde8db6dc94a2c872814b1f16908dee43f43ed\": rpc error: code = NotFound desc = could not find container \"a2775b9ce46bd8bcb696c627a3cde8db6dc94a2c872814b1f16908dee43f43ed\": container with ID starting with a2775b9ce46bd8bcb696c627a3cde8db6dc94a2c872814b1f16908dee43f43ed not found: ID does not exist"
Oct 31 00:55:25.974806 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:25.974338    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl"]
Oct 31 00:55:25.977740 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:25.976507    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bbb902-5bb8dbb74-zr8vl"]
Oct 31 00:55:27.923749 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:27.923672    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1783fef-e971-4cce-861a-37b8c232d0b3" path="/var/lib/kubelet/pods/b1783fef-e971-4cce-861a-37b8c232d0b3/volumes"
Oct 31 00:55:33.663808 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:33.663761    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"]
Oct 31 00:55:33.664307 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:33.664053    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt" podUID="46ed7530-493b-48d4-ac03-a4d9179504ff" containerName="istio-proxy" containerID="cri-o://f158cf25842ee8d3c70bcaa2740451a81bf0475e18550ed9c8c4ee2c936bfdbf" gracePeriod=30
Oct 31 00:55:35.110752 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:35.110710    4327 patch_prober.go:29] interesting pod/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:55:35.111134 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:35.110783    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt" podUID="46ed7530-493b-48d4-ac03-a4d9179504ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:55:39.056767 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.056741    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:55:39.099241 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.099201    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-credential-socket\") pod \"46ed7530-493b-48d4-ac03-a4d9179504ff\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") "
Oct 31 00:55:39.099419 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.099359    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ed7530-493b-48d4-ac03-a4d9179504ff-kube-api-access-79stl\" (UniqueName: \"kubernetes.io/projected/46ed7530-493b-48d4-ac03-a4d9179504ff-kube-api-access-79stl\") pod \"46ed7530-493b-48d4-ac03-a4d9179504ff\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") "
Oct 31 00:55:39.099419 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.099398    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-data\") pod \"46ed7530-493b-48d4-ac03-a4d9179504ff\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") "
Oct 31 00:55:39.099521 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.099475    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-workload-socket\") pod \"46ed7530-493b-48d4-ac03-a4d9179504ff\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") "
Oct 31 00:55:39.099577 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.099537    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-workload-certs\") pod \"46ed7530-493b-48d4-ac03-a4d9179504ff\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") "
Oct 31 00:55:39.099577 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.099539    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-credential-socket" pod "46ed7530-493b-48d4-ac03-a4d9179504ff" (UID: "46ed7530-493b-48d4-ac03-a4d9179504ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:55:39.099577 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.099564    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-envoy\") pod \"46ed7530-493b-48d4-ac03-a4d9179504ff\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") "
Oct 31 00:55:39.099773 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.099626    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-podinfo\") pod \"46ed7530-493b-48d4-ac03-a4d9179504ff\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") "
Oct 31 00:55:39.099773 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.099714    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-token\" (UniqueName: \"kubernetes.io/projected/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-token\") pod \"46ed7530-493b-48d4-ac03-a4d9179504ff\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") "
Oct 31 00:55:39.099773 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.099744    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46ed7530-493b-48d4-ac03-a4d9179504ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ed7530-493b-48d4-ac03-a4d9179504ff-istiod-ca-cert\") pod \"46ed7530-493b-48d4-ac03-a4d9179504ff\" (UID: \"46ed7530-493b-48d4-ac03-a4d9179504ff\") "
Oct 31 00:55:39.099945 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.099806    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-data" pod "46ed7530-493b-48d4-ac03-a4d9179504ff" (UID: "46ed7530-493b-48d4-ac03-a4d9179504ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:55:39.099945 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.099819    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-workload-socket" pod "46ed7530-493b-48d4-ac03-a4d9179504ff" (UID: "46ed7530-493b-48d4-ac03-a4d9179504ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:55:39.099945 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.099836    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-workload-certs" pod "46ed7530-493b-48d4-ac03-a4d9179504ff" (UID: "46ed7530-493b-48d4-ac03-a4d9179504ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:55:39.100227 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.100203    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ed7530-493b-48d4-ac03-a4d9179504ff-istiod-ca-cert" pod "46ed7530-493b-48d4-ac03-a4d9179504ff" (UID: "46ed7530-493b-48d4-ac03-a4d9179504ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:55:39.100749 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.100724    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:55:39.100880 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.100752    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:55:39.100880 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.100767    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:55:39.100880 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.100781    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:55:39.100880 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.100795    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ed7530-493b-48d4-ac03-a4d9179504ff-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:55:39.101900 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.101871    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ed7530-493b-48d4-ac03-a4d9179504ff-kube-api-access-79stl" pod "46ed7530-493b-48d4-ac03-a4d9179504ff" (UID: "46ed7530-493b-48d4-ac03-a4d9179504ff"). InnerVolumeSpecName "kube-api-access-79stl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:55:39.102138 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.102114    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-envoy" pod "46ed7530-493b-48d4-ac03-a4d9179504ff" (UID: "46ed7530-493b-48d4-ac03-a4d9179504ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:55:39.102199 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.102137    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-token" pod "46ed7530-493b-48d4-ac03-a4d9179504ff" (UID: "46ed7530-493b-48d4-ac03-a4d9179504ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:55:39.102578 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.102562    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-podinfo" pod "46ed7530-493b-48d4-ac03-a4d9179504ff" (UID: "46ed7530-493b-48d4-ac03-a4d9179504ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:55:39.104796 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.104772    4327 generic.go:356] "Generic (PLEG): container finished" podID="46ed7530-493b-48d4-ac03-a4d9179504ff" containerID="f158cf25842ee8d3c70bcaa2740451a81bf0475e18550ed9c8c4ee2c936bfdbf" exitCode=0
Oct 31 00:55:39.104898 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.104825    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"
Oct 31 00:55:39.104898 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.104829    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt" event={"ID":"46ed7530-493b-48d4-ac03-a4d9179504ff","Type":"ContainerDied","Data":"f158cf25842ee8d3c70bcaa2740451a81bf0475e18550ed9c8c4ee2c936bfdbf"}
Oct 31 00:55:39.104898 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.104890    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt" event={"ID":"46ed7530-493b-48d4-ac03-a4d9179504ff","Type":"ContainerDied","Data":"96217f6a42053d3f979e847414deb9bf1d66776677fb4ea114ee62886db2b747"}
Oct 31 00:55:39.105070 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.104924    4327 scope.go:122] "RemoveContainer" containerID="f158cf25842ee8d3c70bcaa2740451a81bf0475e18550ed9c8c4ee2c936bfdbf"
Oct 31 00:55:39.147883 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.147855    4327 scope.go:122] "RemoveContainer" containerID="aa239a74c88519d1ce63bd77b96863351fced16f790a1140af92e1b40d72ea9f"
Oct 31 00:55:39.161149 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.161125    4327 scope.go:122] "RemoveContainer" containerID="f158cf25842ee8d3c70bcaa2740451a81bf0475e18550ed9c8c4ee2c936bfdbf"
Oct 31 00:55:39.161561 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:55:39.161527    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f158cf25842ee8d3c70bcaa2740451a81bf0475e18550ed9c8c4ee2c936bfdbf\": container with ID starting with f158cf25842ee8d3c70bcaa2740451a81bf0475e18550ed9c8c4ee2c936bfdbf not found: ID does not exist" containerID="f158cf25842ee8d3c70bcaa2740451a81bf0475e18550ed9c8c4ee2c936bfdbf"
Oct 31 00:55:39.161733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.161574    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f158cf25842ee8d3c70bcaa2740451a81bf0475e18550ed9c8c4ee2c936bfdbf"} err="failed to get container status \"f158cf25842ee8d3c70bcaa2740451a81bf0475e18550ed9c8c4ee2c936bfdbf\": rpc error: code = NotFound desc = could not find container \"f158cf25842ee8d3c70bcaa2740451a81bf0475e18550ed9c8c4ee2c936bfdbf\": container with ID starting with f158cf25842ee8d3c70bcaa2740451a81bf0475e18550ed9c8c4ee2c936bfdbf not found: ID does not exist"
Oct 31 00:55:39.161733 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.161611    4327 scope.go:122] "RemoveContainer" containerID="aa239a74c88519d1ce63bd77b96863351fced16f790a1140af92e1b40d72ea9f"
Oct 31 00:55:39.162011 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:55:39.161977    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa239a74c88519d1ce63bd77b96863351fced16f790a1140af92e1b40d72ea9f\": container with ID starting with aa239a74c88519d1ce63bd77b96863351fced16f790a1140af92e1b40d72ea9f not found: ID does not exist" containerID="aa239a74c88519d1ce63bd77b96863351fced16f790a1140af92e1b40d72ea9f"
Oct 31 00:55:39.162108 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.162017    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa239a74c88519d1ce63bd77b96863351fced16f790a1140af92e1b40d72ea9f"} err="failed to get container status \"aa239a74c88519d1ce63bd77b96863351fced16f790a1140af92e1b40d72ea9f\": rpc error: code = NotFound desc = could not find container \"aa239a74c88519d1ce63bd77b96863351fced16f790a1140af92e1b40d72ea9f\": container with ID starting with aa239a74c88519d1ce63bd77b96863351fced16f790a1140af92e1b40d72ea9f not found: ID does not exist"
Oct 31 00:55:39.189853 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.189762    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"]
Oct 31 00:55:39.203108 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.203078    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79stl\" (UniqueName: \"kubernetes.io/projected/46ed7530-493b-48d4-ac03-a4d9179504ff-kube-api-access-79stl\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:55:39.203108 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.203108    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:55:39.203327 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.203123    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:55:39.203327 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.203134    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ed7530-493b-48d4-ac03-a4d9179504ff-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:55:39.211958 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.211924    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-bbb902-openshift-default-7c646bcb67-wkvqt"]
Oct 31 00:55:39.923825 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:39.923793    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46ed7530-493b-48d4-ac03-a4d9179504ff" path="/var/lib/kubelet/pods/46ed7530-493b-48d4-ac03-a4d9179504ff/volumes"
Oct 31 00:55:41.956272 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:41.956197    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx"]
Oct 31 00:55:41.957017 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:41.957003    4327 cpu_manager.go:414] "containerMap: removing container" podUID="46ed7530-493b-48d4-ac03-a4d9179504ff" containerName="istio-proxy"
Oct 31 00:55:41.957074 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:41.957019    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ed7530-493b-48d4-ac03-a4d9179504ff" containerName="istio-proxy"
Oct 31 00:55:41.957074 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:41.957059    4327 cpu_manager.go:414] "containerMap: removing container" podUID="b1783fef-e971-4cce-861a-37b8c232d0b3" containerName="maas-api"
Oct 31 00:55:41.957074 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:41.957064    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1783fef-e971-4cce-861a-37b8c232d0b3" containerName="maas-api"
Oct 31 00:55:41.957197 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:41.957187    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ed7530-493b-48d4-ac03-a4d9179504ff" containerName="istio-proxy"
Oct 31 00:55:41.957233 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:41.957206    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1783fef-e971-4cce-861a-37b8c232d0b3" containerName="maas-api"
Oct 31 00:55:41.962650 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:41.962627    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx"
Oct 31 00:55:41.985083 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:41.985053    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9de7f5-serving-cert\""
Oct 31 00:55:41.993139 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:41.992146    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx"]
Oct 31 00:55:42.050514 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:42.050474    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3722996-be27-491c-9ee4-62733e3071a9-metrics-certs\") pod \"maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx\" (UID: \"a3722996-be27-491c-9ee4-62733e3071a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx"
Oct 31 00:55:42.050854 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:42.050827    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3722996-be27-491c-9ee4-62733e3071a9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx\" (UID: \"a3722996-be27-491c-9ee4-62733e3071a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx"
Oct 31 00:55:42.050975 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:42.050947    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpk7f\" (UniqueName: \"kubernetes.io/projected/a3722996-be27-491c-9ee4-62733e3071a9-kube-api-access-kpk7f\") pod \"maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx\" (UID: \"a3722996-be27-491c-9ee4-62733e3071a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx"
Oct 31 00:55:42.051043 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:42.051001    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3722996-be27-491c-9ee4-62733e3071a9-maas-api-tls\") pod \"maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx\" (UID: \"a3722996-be27-491c-9ee4-62733e3071a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx"
Oct 31 00:55:42.153271 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:42.153231    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3722996-be27-491c-9ee4-62733e3071a9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx\" (UID: \"a3722996-be27-491c-9ee4-62733e3071a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx"
Oct 31 00:55:42.153488 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:42.153296    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpk7f\" (UniqueName: \"kubernetes.io/projected/a3722996-be27-491c-9ee4-62733e3071a9-kube-api-access-kpk7f\") pod \"maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx\" (UID: \"a3722996-be27-491c-9ee4-62733e3071a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx"
Oct 31 00:55:42.153488 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:42.153325    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3722996-be27-491c-9ee4-62733e3071a9-maas-api-tls\") pod \"maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx\" (UID: \"a3722996-be27-491c-9ee4-62733e3071a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx"
Oct 31 00:55:42.153609 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:42.153509    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3722996-be27-491c-9ee4-62733e3071a9-metrics-certs\") pod \"maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx\" (UID: \"a3722996-be27-491c-9ee4-62733e3071a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx"
Oct 31 00:55:42.154027 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:42.153995    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3722996-be27-491c-9ee4-62733e3071a9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx\" (UID: \"a3722996-be27-491c-9ee4-62733e3071a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx"
Oct 31 00:55:42.155995 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:42.155967    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3722996-be27-491c-9ee4-62733e3071a9-metrics-certs\") pod \"maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx\" (UID: \"a3722996-be27-491c-9ee4-62733e3071a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx"
Oct 31 00:55:42.156101 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:42.156076    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3722996-be27-491c-9ee4-62733e3071a9-maas-api-tls\") pod \"maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx\" (UID: \"a3722996-be27-491c-9ee4-62733e3071a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx"
Oct 31 00:55:42.177878 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:42.177847    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpk7f\" (UniqueName: \"kubernetes.io/projected/a3722996-be27-491c-9ee4-62733e3071a9-kube-api-access-kpk7f\") pod \"maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx\" (UID: \"a3722996-be27-491c-9ee4-62733e3071a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx"
Oct 31 00:55:42.313099 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:42.313023    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx"
Oct 31 00:55:42.681853 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:42.681823    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx"]
Oct 31 00:55:42.682586 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:55:42.682558    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3722996_be27_491c_9ee4_62733e3071a9.slice/crio-54dcc9e4a8b3f931b7a8f103a61d87bb5a29ffecd1f5c523ac7c7a6fa03fe7b6 WatchSource:0}: Error finding container 54dcc9e4a8b3f931b7a8f103a61d87bb5a29ffecd1f5c523ac7c7a6fa03fe7b6: Status 404 returned error can't find the container with id 54dcc9e4a8b3f931b7a8f103a61d87bb5a29ffecd1f5c523ac7c7a6fa03fe7b6
Oct 31 00:55:42.684083 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:42.684059    4327 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:55:43.243877 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:43.243839    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx" event={"ID":"a3722996-be27-491c-9ee4-62733e3071a9","Type":"ContainerStarted","Data":"54dcc9e4a8b3f931b7a8f103a61d87bb5a29ffecd1f5c523ac7c7a6fa03fe7b6"}
Oct 31 00:55:44.257356 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:44.257313    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx" event={"ID":"a3722996-be27-491c-9ee4-62733e3071a9","Type":"ContainerStarted","Data":"7fb36aff183db63cc71b805343608f54fbc3a015539c4f9646e22c555fabb8ad"}
Oct 31 00:55:44.257829 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:44.257410    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx"
Oct 31 00:55:44.302483 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:44.302424    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx" podStartSLOduration=2.7549065969999997 podStartE2EDuration="3.302409382s" totalImagesPullingTime="547.502785ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:44.300426558 +0000 UTC m=+2204.522462101" watchObservedRunningTime="2026-10-31 00:55:44.302409382 +0000 UTC m=+2204.524444924"
Oct 31 00:55:44.969164 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:44.969113    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt"]
Oct 31 00:55:45.321004 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:55:45.320910    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:55:49.067806 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:49.054481    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6"]
Oct 31 00:55:49.067806 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:49.064411    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6" podUID="6c45bbac-3d1d-459e-ab30-811a05b10ea0" containerName="maas-api" containerID="cri-o://451416a13b9d501b2f6975f5677a224b575f4c7b5011d02566ead81fe475c2ee" gracePeriod=30
Oct 31 00:55:49.659712 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:49.657999    4327 generic.go:356] "Generic (PLEG): container finished" podID="6c45bbac-3d1d-459e-ab30-811a05b10ea0" containerID="451416a13b9d501b2f6975f5677a224b575f4c7b5011d02566ead81fe475c2ee" exitCode=0
Oct 31 00:55:49.659712 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:49.658104    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6" event={"ID":"6c45bbac-3d1d-459e-ab30-811a05b10ea0","Type":"ContainerDied","Data":"451416a13b9d501b2f6975f5677a224b575f4c7b5011d02566ead81fe475c2ee"}
Oct 31 00:55:49.767464 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:49.767428    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6"
Oct 31 00:55:49.845179 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:49.841754    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c45bbac-3d1d-459e-ab30-811a05b10ea0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c45bbac-3d1d-459e-ab30-811a05b10ea0-maas-api-tls\") pod \"6c45bbac-3d1d-459e-ab30-811a05b10ea0\" (UID: \"6c45bbac-3d1d-459e-ab30-811a05b10ea0\") "
Oct 31 00:55:49.850387 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:49.846848    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c45bbac-3d1d-459e-ab30-811a05b10ea0-kube-api-access-2x9dq\" (UniqueName: \"kubernetes.io/projected/6c45bbac-3d1d-459e-ab30-811a05b10ea0-kube-api-access-2x9dq\") pod \"6c45bbac-3d1d-459e-ab30-811a05b10ea0\" (UID: \"6c45bbac-3d1d-459e-ab30-811a05b10ea0\") "
Oct 31 00:55:49.850387 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:49.847135    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c45bbac-3d1d-459e-ab30-811a05b10ea0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c45bbac-3d1d-459e-ab30-811a05b10ea0-metrics-certs\") pod \"6c45bbac-3d1d-459e-ab30-811a05b10ea0\" (UID: \"6c45bbac-3d1d-459e-ab30-811a05b10ea0\") "
Oct 31 00:55:49.850387 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:49.847230    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c45bbac-3d1d-459e-ab30-811a05b10ea0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c45bbac-3d1d-459e-ab30-811a05b10ea0-openshift-service-ca\") pod \"6c45bbac-3d1d-459e-ab30-811a05b10ea0\" (UID: \"6c45bbac-3d1d-459e-ab30-811a05b10ea0\") "
Oct 31 00:55:49.893284 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:49.885751    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c45bbac-3d1d-459e-ab30-811a05b10ea0-metrics-certs" pod "6c45bbac-3d1d-459e-ab30-811a05b10ea0" (UID: "6c45bbac-3d1d-459e-ab30-811a05b10ea0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:55:49.893284 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:49.886987    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c45bbac-3d1d-459e-ab30-811a05b10ea0-openshift-service-ca" pod "6c45bbac-3d1d-459e-ab30-811a05b10ea0" (UID: "6c45bbac-3d1d-459e-ab30-811a05b10ea0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:55:49.898253 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:49.894311    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c45bbac-3d1d-459e-ab30-811a05b10ea0-maas-api-tls" pod "6c45bbac-3d1d-459e-ab30-811a05b10ea0" (UID: "6c45bbac-3d1d-459e-ab30-811a05b10ea0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:55:49.898253 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:49.897001    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c45bbac-3d1d-459e-ab30-811a05b10ea0-kube-api-access-2x9dq" pod "6c45bbac-3d1d-459e-ab30-811a05b10ea0" (UID: "6c45bbac-3d1d-459e-ab30-811a05b10ea0"). InnerVolumeSpecName "kube-api-access-2x9dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:55:49.950587 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:49.950547    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c45bbac-3d1d-459e-ab30-811a05b10ea0-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:55:49.951284 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:49.950592    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x9dq\" (UniqueName: \"kubernetes.io/projected/6c45bbac-3d1d-459e-ab30-811a05b10ea0-kube-api-access-2x9dq\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:55:49.951429 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:49.951296    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c45bbac-3d1d-459e-ab30-811a05b10ea0-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:55:49.951429 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:49.951313    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c45bbac-3d1d-459e-ab30-811a05b10ea0-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:55:49.969569 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:49.969522    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt" podUID="31633996-ae18-4e91-bc33-f604afbefade" containerName="payload-pre-processing" containerID="cri-o://018eca14a58d0a6aeed78d5c07c5855bf93f4d54e37b398149c861381829fb77" gracePeriod=25
Oct 31 00:55:50.358155 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:50.358120    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx"
Oct 31 00:55:50.449778 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:50.449742    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt"
Oct 31 00:55:50.546647 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:50.546557    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31633996-ae18-4e91-bc33-f604afbefade-kube-api-access-qbstv\" (UniqueName: \"kubernetes.io/projected/31633996-ae18-4e91-bc33-f604afbefade-kube-api-access-qbstv\") pod \"31633996-ae18-4e91-bc33-f604afbefade\" (UID: \"31633996-ae18-4e91-bc33-f604afbefade\") "
Oct 31 00:55:50.546838 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:50.546710    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31633996-ae18-4e91-bc33-f604afbefade-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31633996-ae18-4e91-bc33-f604afbefade-plugins-config-volume\") pod \"31633996-ae18-4e91-bc33-f604afbefade\" (UID: \"31633996-ae18-4e91-bc33-f604afbefade\") "
Oct 31 00:55:50.547159 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:50.547127    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31633996-ae18-4e91-bc33-f604afbefade-plugins-config-volume" pod "31633996-ae18-4e91-bc33-f604afbefade" (UID: "31633996-ae18-4e91-bc33-f604afbefade"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:55:50.547600 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:50.547577    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31633996-ae18-4e91-bc33-f604afbefade-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:55:50.548920 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:50.548887    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31633996-ae18-4e91-bc33-f604afbefade-kube-api-access-qbstv" pod "31633996-ae18-4e91-bc33-f604afbefade" (UID: "31633996-ae18-4e91-bc33-f604afbefade"). InnerVolumeSpecName "kube-api-access-qbstv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:55:50.648969 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:50.648934    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbstv\" (UniqueName: \"kubernetes.io/projected/31633996-ae18-4e91-bc33-f604afbefade-kube-api-access-qbstv\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:55:50.667736 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:50.667664    4327 generic.go:356] "Generic (PLEG): container finished" podID="31633996-ae18-4e91-bc33-f604afbefade" containerID="018eca14a58d0a6aeed78d5c07c5855bf93f4d54e37b398149c861381829fb77" exitCode=0
Oct 31 00:55:50.667736 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:50.667735    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt"
Oct 31 00:55:50.667978 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:50.667746    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt" event={"ID":"31633996-ae18-4e91-bc33-f604afbefade","Type":"ContainerDied","Data":"018eca14a58d0a6aeed78d5c07c5855bf93f4d54e37b398149c861381829fb77"}
Oct 31 00:55:50.667978 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:50.667808    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt" event={"ID":"31633996-ae18-4e91-bc33-f604afbefade","Type":"ContainerDied","Data":"e80fb03766866cb893ab78e319ad48742f982ef610a330e2e22edca380d7743d"}
Oct 31 00:55:50.667978 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:50.667832    4327 scope.go:122] "RemoveContainer" containerID="018eca14a58d0a6aeed78d5c07c5855bf93f4d54e37b398149c861381829fb77"
Oct 31 00:55:50.669592 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:50.669570    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6"
Oct 31 00:55:50.669821 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:50.669588    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6" event={"ID":"6c45bbac-3d1d-459e-ab30-811a05b10ea0","Type":"ContainerDied","Data":"5af4ce7598cffde9e4ad78f639427b2d690b770565e0e2cd8775deac879b37ac"}
Oct 31 00:55:50.680056 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:50.680040    4327 scope.go:122] "RemoveContainer" containerID="018eca14a58d0a6aeed78d5c07c5855bf93f4d54e37b398149c861381829fb77"
Oct 31 00:55:50.680330 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:55:50.680311    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"018eca14a58d0a6aeed78d5c07c5855bf93f4d54e37b398149c861381829fb77\": container with ID starting with 018eca14a58d0a6aeed78d5c07c5855bf93f4d54e37b398149c861381829fb77 not found: ID does not exist" containerID="018eca14a58d0a6aeed78d5c07c5855bf93f4d54e37b398149c861381829fb77"
Oct 31 00:55:50.680376 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:50.680338    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"018eca14a58d0a6aeed78d5c07c5855bf93f4d54e37b398149c861381829fb77"} err="failed to get container status \"018eca14a58d0a6aeed78d5c07c5855bf93f4d54e37b398149c861381829fb77\": rpc error: code = NotFound desc = could not find container \"018eca14a58d0a6aeed78d5c07c5855bf93f4d54e37b398149c861381829fb77\": container with ID starting with 018eca14a58d0a6aeed78d5c07c5855bf93f4d54e37b398149c861381829fb77 not found: ID does not exist"
Oct 31 00:55:50.680376 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:50.680356    4327 scope.go:122] "RemoveContainer" containerID="451416a13b9d501b2f6975f5677a224b575f4c7b5011d02566ead81fe475c2ee"
Oct 31 00:55:50.726880 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:50.726852    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6"]
Oct 31 00:55:50.731074 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:50.731049    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6"]
Oct 31 00:55:50.768757 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:50.768730    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt"]
Oct 31 00:55:50.778233 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:50.778206    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bdb61f-69cd5d978c-tw6mt"]
Oct 31 00:55:51.923796 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:51.923757    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31633996-ae18-4e91-bc33-f604afbefade" path="/var/lib/kubelet/pods/31633996-ae18-4e91-bc33-f604afbefade/volumes"
Oct 31 00:55:51.924296 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:51.924281    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c45bbac-3d1d-459e-ab30-811a05b10ea0" path="/var/lib/kubelet/pods/6c45bbac-3d1d-459e-ab30-811a05b10ea0/volumes"
Oct 31 00:55:52.679098 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:52.679050    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82d887-d9469d9d9-7skb6" podUID="6c45bbac-3d1d-459e-ab30-811a05b10ea0" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.136:8443/health\": context deadline exceeded"
Oct 31 00:55:58.305603 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:58.301330    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"]
Oct 31 00:55:58.305603 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:58.301608    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97" podUID="cb121139-61a4-498c-9ace-56d1394a8205" containerName="istio-proxy" containerID="cri-o://410eb56ece4aeb791fb8c447b972e43e90ab870f7329a9880661470a56e0d1df" gracePeriod=30
Oct 31 00:55:58.305603 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:58.305534    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"]
Oct 31 00:55:58.306402 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:58.305777    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp" podUID="4511417e-7a7f-4671-969b-b87f7668c3f4" containerName="istio-proxy" containerID="cri-o://67323dbef922683532bd04de5e623c15f47f9867c5a31a275dc588e6c610cf42" gracePeriod=30
Oct 31 00:55:59.729198 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.729162    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"]
Oct 31 00:55:59.730431 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.730405    4327 cpu_manager.go:414] "containerMap: removing container" podUID="31633996-ae18-4e91-bc33-f604afbefade" containerName="payload-pre-processing"
Oct 31 00:55:59.730533 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.730436    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31633996-ae18-4e91-bc33-f604afbefade" containerName="payload-pre-processing"
Oct 31 00:55:59.730533 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.730521    4327 cpu_manager.go:414] "containerMap: removing container" podUID="46ed7530-493b-48d4-ac03-a4d9179504ff" containerName="istio-proxy"
Oct 31 00:55:59.730533 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.730529    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ed7530-493b-48d4-ac03-a4d9179504ff" containerName="istio-proxy"
Oct 31 00:55:59.730633 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.730556    4327 cpu_manager.go:414] "containerMap: removing container" podUID="6c45bbac-3d1d-459e-ab30-811a05b10ea0" containerName="maas-api"
Oct 31 00:55:59.730633 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.730564    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c45bbac-3d1d-459e-ab30-811a05b10ea0" containerName="maas-api"
Oct 31 00:55:59.730797 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.730780    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ed7530-493b-48d4-ac03-a4d9179504ff" containerName="istio-proxy"
Oct 31 00:55:59.730850 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.730808    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c45bbac-3d1d-459e-ab30-811a05b10ea0" containerName="maas-api"
Oct 31 00:55:59.730850 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.730827    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="31633996-ae18-4e91-bc33-f604afbefade" containerName="payload-pre-processing"
Oct 31 00:55:59.735888 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.735862    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:55:59.766251 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.766220    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-57e594-openshift-default-dockercfg-4ch9j\""
Oct 31 00:55:59.785295 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.785263    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"]
Oct 31 00:55:59.799695 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.799646    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istiod-ca-cert\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:55:59.799899 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.799721    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-credential-socket\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:55:59.799899 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.799757    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-token\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:55:59.799899 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.799807    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-workload-certs\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:55:59.800089 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.800074    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlvx8\" (UniqueName: \"kubernetes.io/projected/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-kube-api-access-wlvx8\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:55:59.800170 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.800151    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-data\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:55:59.800271 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.800259    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-workload-socket\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:55:59.800326 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.800279    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-envoy\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:55:59.800326 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.800299    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-podinfo\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:55:59.902002 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.901965    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-credential-socket\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:55:59.902002 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.902004    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-token\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:55:59.902224 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.902034    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-workload-certs\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:55:59.902224 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.902182    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlvx8\" (UniqueName: \"kubernetes.io/projected/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-kube-api-access-wlvx8\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:55:59.902311 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.902245    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-data\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:55:59.902402 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.902381    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-credential-socket\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:55:59.902402 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.902394    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-workload-socket\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:55:59.902484 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.902426    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-envoy\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:55:59.902484 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.902458    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-podinfo\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:55:59.902484 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.902469    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-workload-certs\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:55:59.902602 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.902579    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istiod-ca-cert\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:55:59.902602 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.902581    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-data\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:55:59.902716 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.902625    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-workload-socket\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:55:59.903200 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.903177    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istiod-ca-cert\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:55:59.904626 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.904601    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-envoy\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:55:59.904944 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:55:59.904925    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-podinfo\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:56:00.060755 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:00.006566    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlvx8\" (UniqueName: \"kubernetes.io/projected/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-kube-api-access-wlvx8\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:56:00.095723 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:00.089484    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-token\") pod \"e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") " pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:56:00.368903 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:00.368872    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-57e594-openshift-default-dockercfg-4ch9j\""
Oct 31 00:56:00.374994 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:00.374970    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:56:00.708822 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:00.708790    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"]
Oct 31 00:56:00.710401 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:56:00.710375    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f13ccf_97ff_4e16_ba0b_c050043ce8c8.slice/crio-43b7b0848c285a2bdc7d5cc553530799a66902a48aee11b4929d88a3c4a58cab WatchSource:0}: Error finding container 43b7b0848c285a2bdc7d5cc553530799a66902a48aee11b4929d88a3c4a58cab: Status 404 returned error can't find the container with id 43b7b0848c285a2bdc7d5cc553530799a66902a48aee11b4929d88a3c4a58cab
Oct 31 00:56:00.713437 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:00.713396    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:56:00.713548 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:00.713468    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:56:00.713548 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:00.713497    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:56:00.847560 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:00.847483    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9" event={"ID":"20f13ccf-97ff-4e16-ba0b-c050043ce8c8","Type":"ContainerStarted","Data":"2ace50a97b7ca77da7c480b71ce106487ced0112d72fcb42d69e640c98049cad"}
Oct 31 00:56:00.847560 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:00.847554    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9" event={"ID":"20f13ccf-97ff-4e16-ba0b-c050043ce8c8","Type":"ContainerStarted","Data":"43b7b0848c285a2bdc7d5cc553530799a66902a48aee11b4929d88a3c4a58cab"}
Oct 31 00:56:00.904140 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:00.904079    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9" podStartSLOduration=1.904064945 podStartE2EDuration="1.904064945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:00.901236282 +0000 UTC m=+2221.123271854" watchObservedRunningTime="2026-10-31 00:56:00.904064945 +0000 UTC m=+2221.126100561"
Oct 31 00:56:01.375690 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:01.375644    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:56:01.382372 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:01.382345    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:56:01.856346 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:01.856319    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:56:01.857574 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:01.857543    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:56:03.177037 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.177005    4327 scope.go:122] "RemoveContainer" containerID="ce23b983f8a1be07e3e118480789461c4bc345b4404284a7c7405c2b3609a105"
Oct 31 00:56:03.660368 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.660338    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:56:03.664912 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.664888    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:56:03.822515 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.822428    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb121139-61a4-498c-9ace-56d1394a8205-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb121139-61a4-498c-9ace-56d1394a8205-istio-podinfo\") pod \"cb121139-61a4-498c-9ace-56d1394a8205\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") "
Oct 31 00:56:03.822515 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.822505    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-token\" (UniqueName: \"kubernetes.io/projected/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-token\") pod \"4511417e-7a7f-4671-969b-b87f7668c3f4\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") "
Oct 31 00:56:03.822787 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.822562    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4511417e-7a7f-4671-969b-b87f7668c3f4-kube-api-access-d9tmj\" (UniqueName: \"kubernetes.io/projected/4511417e-7a7f-4671-969b-b87f7668c3f4-kube-api-access-d9tmj\") pod \"4511417e-7a7f-4671-969b-b87f7668c3f4\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") "
Oct 31 00:56:03.822787 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.822614    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-data\") pod \"4511417e-7a7f-4671-969b-b87f7668c3f4\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") "
Oct 31 00:56:03.822787 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.822692    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-istio-envoy\") pod \"cb121139-61a4-498c-9ace-56d1394a8205\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") "
Oct 31 00:56:03.822787 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.822752    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb121139-61a4-498c-9ace-56d1394a8205-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb121139-61a4-498c-9ace-56d1394a8205-istiod-ca-cert\") pod \"cb121139-61a4-498c-9ace-56d1394a8205\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") "
Oct 31 00:56:03.823003 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.822803    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-workload-certs\") pod \"4511417e-7a7f-4671-969b-b87f7668c3f4\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") "
Oct 31 00:56:03.823003 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.822854    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-podinfo\") pod \"4511417e-7a7f-4671-969b-b87f7668c3f4\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") "
Oct 31 00:56:03.823003 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.822917    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-workload-certs\") pod \"cb121139-61a4-498c-9ace-56d1394a8205\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") "
Oct 31 00:56:03.823003 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.822958    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb121139-61a4-498c-9ace-56d1394a8205-kube-api-access-8wqqc\" (UniqueName: \"kubernetes.io/projected/cb121139-61a4-498c-9ace-56d1394a8205-kube-api-access-8wqqc\") pod \"cb121139-61a4-498c-9ace-56d1394a8205\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") "
Oct 31 00:56:03.823219 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.823013    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-credential-socket\") pod \"4511417e-7a7f-4671-969b-b87f7668c3f4\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") "
Oct 31 00:56:03.823219 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.823062    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-credential-socket\") pod \"cb121139-61a4-498c-9ace-56d1394a8205\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") "
Oct 31 00:56:03.823219 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.823122    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb121139-61a4-498c-9ace-56d1394a8205-istio-token\" (UniqueName: \"kubernetes.io/projected/cb121139-61a4-498c-9ace-56d1394a8205-istio-token\") pod \"cb121139-61a4-498c-9ace-56d1394a8205\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") "
Oct 31 00:56:03.823219 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.823157    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-workload-socket\") pod \"4511417e-7a7f-4671-969b-b87f7668c3f4\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") "
Oct 31 00:56:03.823428 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.823294    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-envoy\") pod \"4511417e-7a7f-4671-969b-b87f7668c3f4\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") "
Oct 31 00:56:03.823428 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.823330    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-istio-data\") pod \"cb121139-61a4-498c-9ace-56d1394a8205\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") "
Oct 31 00:56:03.823428 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.823379    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-workload-certs" pod "4511417e-7a7f-4671-969b-b87f7668c3f4" (UID: "4511417e-7a7f-4671-969b-b87f7668c3f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:56:03.823428 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.823387    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-workload-socket\") pod \"cb121139-61a4-498c-9ace-56d1394a8205\" (UID: \"cb121139-61a4-498c-9ace-56d1394a8205\") "
Oct 31 00:56:03.823629 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.823496    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4511417e-7a7f-4671-969b-b87f7668c3f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4511417e-7a7f-4671-969b-b87f7668c3f4-istiod-ca-cert\") pod \"4511417e-7a7f-4671-969b-b87f7668c3f4\" (UID: \"4511417e-7a7f-4671-969b-b87f7668c3f4\") "
Oct 31 00:56:03.823629 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.823613    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-workload-socket" pod "cb121139-61a4-498c-9ace-56d1394a8205" (UID: "cb121139-61a4-498c-9ace-56d1394a8205"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:56:03.823755 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.823624    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-data" pod "4511417e-7a7f-4671-969b-b87f7668c3f4" (UID: "4511417e-7a7f-4671-969b-b87f7668c3f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:56:03.823906 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.823880    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-credential-socket" pod "4511417e-7a7f-4671-969b-b87f7668c3f4" (UID: "4511417e-7a7f-4671-969b-b87f7668c3f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:56:03.824466 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.824065    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-credential-socket" pod "cb121139-61a4-498c-9ace-56d1394a8205" (UID: "cb121139-61a4-498c-9ace-56d1394a8205"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:56:03.824466 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.824213    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-workload-socket" pod "4511417e-7a7f-4671-969b-b87f7668c3f4" (UID: "4511417e-7a7f-4671-969b-b87f7668c3f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:56:03.824466 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.824261    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-workload-certs" pod "cb121139-61a4-498c-9ace-56d1394a8205" (UID: "cb121139-61a4-498c-9ace-56d1394a8205"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:56:03.824776 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.824638    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-istio-data" pod "cb121139-61a4-498c-9ace-56d1394a8205" (UID: "cb121139-61a4-498c-9ace-56d1394a8205"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:56:03.825054 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.825017    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4511417e-7a7f-4671-969b-b87f7668c3f4-istiod-ca-cert" pod "4511417e-7a7f-4671-969b-b87f7668c3f4" (UID: "4511417e-7a7f-4671-969b-b87f7668c3f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:56:03.825054 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.825044    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:56:03.825226 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.825071    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:56:03.825226 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.825088    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:56:03.825226 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.825103    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:56:03.825226 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.825117    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:56:03.825226 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.825131    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:56:03.825226 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.825147    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:56:03.825226 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.825160    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:56:03.825591 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.825308    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb121139-61a4-498c-9ace-56d1394a8205-istiod-ca-cert" pod "cb121139-61a4-498c-9ace-56d1394a8205" (UID: "cb121139-61a4-498c-9ace-56d1394a8205"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:56:03.826177 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.826149    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb121139-61a4-498c-9ace-56d1394a8205-istio-podinfo" pod "cb121139-61a4-498c-9ace-56d1394a8205" (UID: "cb121139-61a4-498c-9ace-56d1394a8205"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:56:03.828619 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.827372    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-envoy" pod "4511417e-7a7f-4671-969b-b87f7668c3f4" (UID: "4511417e-7a7f-4671-969b-b87f7668c3f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:56:03.828619 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.827491    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-podinfo" pod "4511417e-7a7f-4671-969b-b87f7668c3f4" (UID: "4511417e-7a7f-4671-969b-b87f7668c3f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:56:03.828619 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.827641    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-token" pod "4511417e-7a7f-4671-969b-b87f7668c3f4" (UID: "4511417e-7a7f-4671-969b-b87f7668c3f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:56:03.828875 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.828717    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4511417e-7a7f-4671-969b-b87f7668c3f4-kube-api-access-d9tmj" pod "4511417e-7a7f-4671-969b-b87f7668c3f4" (UID: "4511417e-7a7f-4671-969b-b87f7668c3f4"). InnerVolumeSpecName "kube-api-access-d9tmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:56:03.829035 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.829012    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-istio-envoy" pod "cb121139-61a4-498c-9ace-56d1394a8205" (UID: "cb121139-61a4-498c-9ace-56d1394a8205"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:56:03.829647 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.829623    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb121139-61a4-498c-9ace-56d1394a8205-kube-api-access-8wqqc" pod "cb121139-61a4-498c-9ace-56d1394a8205" (UID: "cb121139-61a4-498c-9ace-56d1394a8205"). InnerVolumeSpecName "kube-api-access-8wqqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:56:03.830375 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.830354    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb121139-61a4-498c-9ace-56d1394a8205-istio-token" pod "cb121139-61a4-498c-9ace-56d1394a8205" (UID: "cb121139-61a4-498c-9ace-56d1394a8205"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:56:03.877024 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.876984    4327 generic.go:356] "Generic (PLEG): container finished" podID="4511417e-7a7f-4671-969b-b87f7668c3f4" containerID="67323dbef922683532bd04de5e623c15f47f9867c5a31a275dc588e6c610cf42" exitCode=0
Oct 31 00:56:03.877223 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.877031    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp" event={"ID":"4511417e-7a7f-4671-969b-b87f7668c3f4","Type":"ContainerDied","Data":"67323dbef922683532bd04de5e623c15f47f9867c5a31a275dc588e6c610cf42"}
Oct 31 00:56:03.877223 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.877061    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"
Oct 31 00:56:03.877223 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.877104    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp" event={"ID":"4511417e-7a7f-4671-969b-b87f7668c3f4","Type":"ContainerDied","Data":"3fcb17c7d274cb96bb0ad96f6120943cfd01c77005e1945c18db73f32e8e7e77"}
Oct 31 00:56:03.877223 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.877138    4327 scope.go:122] "RemoveContainer" containerID="67323dbef922683532bd04de5e623c15f47f9867c5a31a275dc588e6c610cf42"
Oct 31 00:56:03.882704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.880387    4327 generic.go:356] "Generic (PLEG): container finished" podID="cb121139-61a4-498c-9ace-56d1394a8205" containerID="410eb56ece4aeb791fb8c447b972e43e90ab870f7329a9880661470a56e0d1df" exitCode=0
Oct 31 00:56:03.882704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.881415    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"
Oct 31 00:56:03.883777 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.883750    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97" event={"ID":"cb121139-61a4-498c-9ace-56d1394a8205","Type":"ContainerDied","Data":"410eb56ece4aeb791fb8c447b972e43e90ab870f7329a9880661470a56e0d1df"}
Oct 31 00:56:03.883886 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.883816    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97" event={"ID":"cb121139-61a4-498c-9ace-56d1394a8205","Type":"ContainerDied","Data":"dbe9564d0713889cbdb389f80979aec6fe54e201826a1e32a05e875cfb2c5da8"}
Oct 31 00:56:03.898171 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.898153    4327 scope.go:122] "RemoveContainer" containerID="67323dbef922683532bd04de5e623c15f47f9867c5a31a275dc588e6c610cf42"
Oct 31 00:56:03.898469 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:56:03.898451    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67323dbef922683532bd04de5e623c15f47f9867c5a31a275dc588e6c610cf42\": container with ID starting with 67323dbef922683532bd04de5e623c15f47f9867c5a31a275dc588e6c610cf42 not found: ID does not exist" containerID="67323dbef922683532bd04de5e623c15f47f9867c5a31a275dc588e6c610cf42"
Oct 31 00:56:03.898557 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.898483    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67323dbef922683532bd04de5e623c15f47f9867c5a31a275dc588e6c610cf42"} err="failed to get container status \"67323dbef922683532bd04de5e623c15f47f9867c5a31a275dc588e6c610cf42\": rpc error: code = NotFound desc = could not find container \"67323dbef922683532bd04de5e623c15f47f9867c5a31a275dc588e6c610cf42\": container with ID starting with 67323dbef922683532bd04de5e623c15f47f9867c5a31a275dc588e6c610cf42 not found: ID does not exist"
Oct 31 00:56:03.898557 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.898506    4327 scope.go:122] "RemoveContainer" containerID="410eb56ece4aeb791fb8c447b972e43e90ab870f7329a9880661470a56e0d1df"
Oct 31 00:56:03.908997 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.908969    4327 scope.go:122] "RemoveContainer" containerID="410eb56ece4aeb791fb8c447b972e43e90ab870f7329a9880661470a56e0d1df"
Oct 31 00:56:03.909289 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:56:03.909262    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"410eb56ece4aeb791fb8c447b972e43e90ab870f7329a9880661470a56e0d1df\": container with ID starting with 410eb56ece4aeb791fb8c447b972e43e90ab870f7329a9880661470a56e0d1df not found: ID does not exist" containerID="410eb56ece4aeb791fb8c447b972e43e90ab870f7329a9880661470a56e0d1df"
Oct 31 00:56:03.909344 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.909298    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"410eb56ece4aeb791fb8c447b972e43e90ab870f7329a9880661470a56e0d1df"} err="failed to get container status \"410eb56ece4aeb791fb8c447b972e43e90ab870f7329a9880661470a56e0d1df\": rpc error: code = NotFound desc = could not find container \"410eb56ece4aeb791fb8c447b972e43e90ab870f7329a9880661470a56e0d1df\": container with ID starting with 410eb56ece4aeb791fb8c447b972e43e90ab870f7329a9880661470a56e0d1df not found: ID does not exist"
Oct 31 00:56:03.926479 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.926452    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:56:03.926479 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.926478    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9tmj\" (UniqueName: \"kubernetes.io/projected/4511417e-7a7f-4671-969b-b87f7668c3f4-kube-api-access-d9tmj\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:56:03.926652 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.926488    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb121139-61a4-498c-9ace-56d1394a8205-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:56:03.926652 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.926498    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb121139-61a4-498c-9ace-56d1394a8205-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:56:03.926652 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.926507    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:56:03.926652 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.926515    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wqqc\" (UniqueName: \"kubernetes.io/projected/cb121139-61a4-498c-9ace-56d1394a8205-kube-api-access-8wqqc\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:56:03.926652 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.926523    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb121139-61a4-498c-9ace-56d1394a8205-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:56:03.926652 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.926532    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4511417e-7a7f-4671-969b-b87f7668c3f4-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:56:03.926652 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.926539    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4511417e-7a7f-4671-969b-b87f7668c3f4-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:56:03.926652 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.926547    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb121139-61a4-498c-9ace-56d1394a8205-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:56:03.955186 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.955137    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"]
Oct 31 00:56:03.959997 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:03.959969    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jdktp"]
Oct 31 00:56:04.017974 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:04.017929    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"]
Oct 31 00:56:04.022052 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:04.022023    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-dxv97"]
Oct 31 00:56:05.291823 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.291789    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2"]
Oct 31 00:56:05.292863 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.292840    4327 cpu_manager.go:414] "containerMap: removing container" podUID="cb121139-61a4-498c-9ace-56d1394a8205" containerName="istio-proxy"
Oct 31 00:56:05.292863 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.292863    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb121139-61a4-498c-9ace-56d1394a8205" containerName="istio-proxy"
Oct 31 00:56:05.293053 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.292879    4327 cpu_manager.go:414] "containerMap: removing container" podUID="4511417e-7a7f-4671-969b-b87f7668c3f4" containerName="istio-proxy"
Oct 31 00:56:05.293053 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.292886    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4511417e-7a7f-4671-969b-b87f7668c3f4" containerName="istio-proxy"
Oct 31 00:56:05.293152 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.293097    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb121139-61a4-498c-9ace-56d1394a8205" containerName="istio-proxy"
Oct 31 00:56:05.293152 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.293111    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="4511417e-7a7f-4671-969b-b87f7668c3f4" containerName="istio-proxy"
Oct 31 00:56:05.298488 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.298467    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2"
Oct 31 00:56:05.318970 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.318941    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2"]
Oct 31 00:56:05.325666 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.325642    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-57e594-dockercfg-t25d7\""
Oct 31 00:56:05.325666 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.325646    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-57e594\""
Oct 31 00:56:05.524139 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.465716    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2"]
Oct 31 00:56:05.524139 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.484469    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2"
Oct 31 00:56:05.547138 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.547056    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2"]
Oct 31 00:56:05.569780 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.566768    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e47fba4e-8b03-401e-9967-c6bdf6c78ec5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2\" (UID: \"e47fba4e-8b03-401e-9967-c6bdf6c78ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2"
Oct 31 00:56:05.569780 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.566868    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw9j2\" (UniqueName: \"kubernetes.io/projected/e47fba4e-8b03-401e-9967-c6bdf6c78ec5-kube-api-access-vw9j2\") pod \"payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2\" (UID: \"e47fba4e-8b03-401e-9967-c6bdf6c78ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2"
Oct 31 00:56:05.670720 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.669228    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkhkf\" (UniqueName: \"kubernetes.io/projected/ed89b1a6-e362-43a8-bca0-1566333e8e9b-kube-api-access-wkhkf\") pod \"payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2\" (UID: \"ed89b1a6-e362-43a8-bca0-1566333e8e9b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2"
Oct 31 00:56:05.670720 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.669461    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e47fba4e-8b03-401e-9967-c6bdf6c78ec5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2\" (UID: \"e47fba4e-8b03-401e-9967-c6bdf6c78ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2"
Oct 31 00:56:05.670720 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.669510    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed89b1a6-e362-43a8-bca0-1566333e8e9b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2\" (UID: \"ed89b1a6-e362-43a8-bca0-1566333e8e9b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2"
Oct 31 00:56:05.670720 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.669561    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw9j2\" (UniqueName: \"kubernetes.io/projected/e47fba4e-8b03-401e-9967-c6bdf6c78ec5-kube-api-access-vw9j2\") pod \"payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2\" (UID: \"e47fba4e-8b03-401e-9967-c6bdf6c78ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2"
Oct 31 00:56:05.670720 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.670643    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e47fba4e-8b03-401e-9967-c6bdf6c78ec5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2\" (UID: \"e47fba4e-8b03-401e-9967-c6bdf6c78ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2"
Oct 31 00:56:05.695335 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.695258    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw9j2\" (UniqueName: \"kubernetes.io/projected/e47fba4e-8b03-401e-9967-c6bdf6c78ec5-kube-api-access-vw9j2\") pod \"payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2\" (UID: \"e47fba4e-8b03-401e-9967-c6bdf6c78ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2"
Oct 31 00:56:05.771533 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.771499    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkhkf\" (UniqueName: \"kubernetes.io/projected/ed89b1a6-e362-43a8-bca0-1566333e8e9b-kube-api-access-wkhkf\") pod \"payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2\" (UID: \"ed89b1a6-e362-43a8-bca0-1566333e8e9b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2"
Oct 31 00:56:05.771759 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.771669    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed89b1a6-e362-43a8-bca0-1566333e8e9b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2\" (UID: \"ed89b1a6-e362-43a8-bca0-1566333e8e9b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2"
Oct 31 00:56:05.772397 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.772376    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed89b1a6-e362-43a8-bca0-1566333e8e9b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2\" (UID: \"ed89b1a6-e362-43a8-bca0-1566333e8e9b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2"
Oct 31 00:56:05.795305 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.795268    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkhkf\" (UniqueName: \"kubernetes.io/projected/ed89b1a6-e362-43a8-bca0-1566333e8e9b-kube-api-access-wkhkf\") pod \"payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2\" (UID: \"ed89b1a6-e362-43a8-bca0-1566333e8e9b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2"
Oct 31 00:56:05.914133 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.914097    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2"
Oct 31 00:56:05.924164 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.924136    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4511417e-7a7f-4671-969b-b87f7668c3f4" path="/var/lib/kubelet/pods/4511417e-7a7f-4671-969b-b87f7668c3f4/volumes"
Oct 31 00:56:05.924714 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:05.924668    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb121139-61a4-498c-9ace-56d1394a8205" path="/var/lib/kubelet/pods/cb121139-61a4-498c-9ace-56d1394a8205/volumes"
Oct 31 00:56:06.008148 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:06.008109    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2"
Oct 31 00:56:06.258340 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:06.253003    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2"]
Oct 31 00:56:06.360586 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:06.360561    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2"]
Oct 31 00:56:06.360920 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:56:06.360617    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded89b1a6_e362_43a8_bca0_1566333e8e9b.slice/crio-ce93c6913f3cd06b75663163332c13e19d0180e0e869191f85223b0b8c6c9562 WatchSource:0}: Error finding container ce93c6913f3cd06b75663163332c13e19d0180e0e869191f85223b0b8c6c9562: Status 404 returned error can't find the container with id ce93c6913f3cd06b75663163332c13e19d0180e0e869191f85223b0b8c6c9562
Oct 31 00:56:06.907013 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:06.906976    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2" event={"ID":"ed89b1a6-e362-43a8-bca0-1566333e8e9b","Type":"ContainerStarted","Data":"c059dbbacb3b7f6aa23de561a58da5e153d0cb356e85955f157ada9f5464b1e4"}
Oct 31 00:56:06.907212 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:06.907039    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2" event={"ID":"ed89b1a6-e362-43a8-bca0-1566333e8e9b","Type":"ContainerStarted","Data":"ce93c6913f3cd06b75663163332c13e19d0180e0e869191f85223b0b8c6c9562"}
Oct 31 00:56:06.907212 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:06.907071    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2"
Oct 31 00:56:06.908826 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:06.908791    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2" event={"ID":"e47fba4e-8b03-401e-9967-c6bdf6c78ec5","Type":"ContainerStarted","Data":"24ca92ad004132022e6293a67c2e0e39a4fce67ff59b0c324d3c14b95163186f"}
Oct 31 00:56:06.908943 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:06.908846    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2"
Oct 31 00:56:06.908943 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:06.908865    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2" event={"ID":"e47fba4e-8b03-401e-9967-c6bdf6c78ec5","Type":"ContainerStarted","Data":"973dd86705bc6f8037413bb6ceb61301b19387b81eb2525c62c34ac5b40fa9e4"}
Oct 31 00:56:06.951728 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:06.951662    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2" podStartSLOduration=1.9516475739999999 podStartE2EDuration="1.951647574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:06.948922273 +0000 UTC m=+2227.170957817" watchObservedRunningTime="2026-10-31 00:56:06.951647574 +0000 UTC m=+2227.173683115"
Oct 31 00:56:06.994190 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:06.994142    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2" podStartSLOduration=1.994128001 podStartE2EDuration="1.994128001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:06.990093325 +0000 UTC m=+2227.212128867" watchObservedRunningTime="2026-10-31 00:56:06.994128001 +0000 UTC m=+2227.216163542"
Oct 31 00:56:15.295158 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:56:15.295118    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:56:17.924063 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:17.924034    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2"
Oct 31 00:56:17.924441 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:17.924092    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2"
Oct 31 00:56:40.843701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:40.843470    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng"]
Oct 31 00:56:40.888703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:40.869433    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng"
Oct 31 00:56:40.888703 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:40.886310    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-82b5e8\""
Oct 31 00:56:40.989698 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:40.988870    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-82b5e8-dockercfg-55hrf\""
Oct 31 00:56:41.116153 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:41.116053    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng"]
Oct 31 00:56:41.126725 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:41.117747    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwt9d\" (UniqueName: \"kubernetes.io/projected/2453aadb-8cf3-403f-b728-d4de97020398-kube-api-access-pwt9d\") pod \"payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng\" (UID: \"2453aadb-8cf3-403f-b728-d4de97020398\") " pod="openshift-ingress/payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng"
Oct 31 00:56:41.126725 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:41.118214    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2453aadb-8cf3-403f-b728-d4de97020398-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng\" (UID: \"2453aadb-8cf3-403f-b728-d4de97020398\") " pod="openshift-ingress/payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng"
Oct 31 00:56:41.171705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:41.161959    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8"]
Oct 31 00:56:41.253725 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:41.247727    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2453aadb-8cf3-403f-b728-d4de97020398-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng\" (UID: \"2453aadb-8cf3-403f-b728-d4de97020398\") " pod="openshift-ingress/payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng"
Oct 31 00:56:41.253725 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:41.247895    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwt9d\" (UniqueName: \"kubernetes.io/projected/2453aadb-8cf3-403f-b728-d4de97020398-kube-api-access-pwt9d\") pod \"payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng\" (UID: \"2453aadb-8cf3-403f-b728-d4de97020398\") " pod="openshift-ingress/payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng"
Oct 31 00:56:41.253725 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:41.249291    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2453aadb-8cf3-403f-b728-d4de97020398-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng\" (UID: \"2453aadb-8cf3-403f-b728-d4de97020398\") " pod="openshift-ingress/payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng"
Oct 31 00:56:41.289539 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:41.283978    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8"]
Oct 31 00:56:41.340797 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:41.289953    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8"
Oct 31 00:56:41.398722 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:41.386073    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5r5q\" (UniqueName: \"kubernetes.io/projected/fd9b6201-5de2-4147-94c2-edd87d1ec204-kube-api-access-w5r5q\") pod \"payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8\" (UID: \"fd9b6201-5de2-4147-94c2-edd87d1ec204\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8"
Oct 31 00:56:41.398722 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:41.386330    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd9b6201-5de2-4147-94c2-edd87d1ec204-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8\" (UID: \"fd9b6201-5de2-4147-94c2-edd87d1ec204\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8"
Oct 31 00:56:41.398722 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:41.390632    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwt9d\" (UniqueName: \"kubernetes.io/projected/2453aadb-8cf3-403f-b728-d4de97020398-kube-api-access-pwt9d\") pod \"payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng\" (UID: \"2453aadb-8cf3-403f-b728-d4de97020398\") " pod="openshift-ingress/payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng"
Oct 31 00:56:41.423578 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:41.419882    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng"
Oct 31 00:56:41.488534 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:41.488498    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd9b6201-5de2-4147-94c2-edd87d1ec204-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8\" (UID: \"fd9b6201-5de2-4147-94c2-edd87d1ec204\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8"
Oct 31 00:56:41.488897 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:41.488871    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5r5q\" (UniqueName: \"kubernetes.io/projected/fd9b6201-5de2-4147-94c2-edd87d1ec204-kube-api-access-w5r5q\") pod \"payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8\" (UID: \"fd9b6201-5de2-4147-94c2-edd87d1ec204\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8"
Oct 31 00:56:41.489338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:41.489318    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd9b6201-5de2-4147-94c2-edd87d1ec204-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8\" (UID: \"fd9b6201-5de2-4147-94c2-edd87d1ec204\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8"
Oct 31 00:56:41.512507 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:41.512476    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5r5q\" (UniqueName: \"kubernetes.io/projected/fd9b6201-5de2-4147-94c2-edd87d1ec204-kube-api-access-w5r5q\") pod \"payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8\" (UID: \"fd9b6201-5de2-4147-94c2-edd87d1ec204\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8"
Oct 31 00:56:41.766344 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:41.766262    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8"
Oct 31 00:56:41.790397 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:41.790363    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng"]
Oct 31 00:56:41.797100 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:56:41.797063    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2453aadb_8cf3_403f_b728_d4de97020398.slice/crio-715c753c24be6251a9b6dc5991d3766fe7e2d4962f9619a7fb547eb976f58fb8 WatchSource:0}: Error finding container 715c753c24be6251a9b6dc5991d3766fe7e2d4962f9619a7fb547eb976f58fb8: Status 404 returned error can't find the container with id 715c753c24be6251a9b6dc5991d3766fe7e2d4962f9619a7fb547eb976f58fb8
Oct 31 00:56:42.146344 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:42.146311    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8"]
Oct 31 00:56:42.147367 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:56:42.147345    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd9b6201_5de2_4147_94c2_edd87d1ec204.slice/crio-10f8da7945a736c59414fddc2d925dd53b9de0bef92499022437ba526dbf790c WatchSource:0}: Error finding container 10f8da7945a736c59414fddc2d925dd53b9de0bef92499022437ba526dbf790c: Status 404 returned error can't find the container with id 10f8da7945a736c59414fddc2d925dd53b9de0bef92499022437ba526dbf790c
Oct 31 00:56:42.382948 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:42.382908    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng" event={"ID":"2453aadb-8cf3-403f-b728-d4de97020398","Type":"ContainerStarted","Data":"7cd05e2f48cae0f1839af8c672c83faa72cfb02f20a402ee124adab5523f43e7"}
Oct 31 00:56:42.383138 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:42.382972    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng" event={"ID":"2453aadb-8cf3-403f-b728-d4de97020398","Type":"ContainerStarted","Data":"715c753c24be6251a9b6dc5991d3766fe7e2d4962f9619a7fb547eb976f58fb8"}
Oct 31 00:56:42.383138 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:42.382999    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng"
Oct 31 00:56:42.384930 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:42.384900    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8" event={"ID":"fd9b6201-5de2-4147-94c2-edd87d1ec204","Type":"ContainerStarted","Data":"8cb7195d654b8bf72ab1e1297e4ae887c8ecfa2cd5cdab1a7e9ed75c25aec6db"}
Oct 31 00:56:42.385064 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:42.384957    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8" event={"ID":"fd9b6201-5de2-4147-94c2-edd87d1ec204","Type":"ContainerStarted","Data":"10f8da7945a736c59414fddc2d925dd53b9de0bef92499022437ba526dbf790c"}
Oct 31 00:56:42.385064 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:42.384987    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8"
Oct 31 00:56:42.469264 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:42.469145    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng" podStartSLOduration=2.469119707 podStartE2EDuration="2.469119707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:42.448090453 +0000 UTC m=+2262.670125995" watchObservedRunningTime="2026-10-31 00:56:42.469119707 +0000 UTC m=+2262.691155249"
Oct 31 00:56:45.448701 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:56:45.444752    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:56:50.492649 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:50.492547    4327 generic.go:356] "Generic (PLEG): container finished" podID="b046f39f-ed7f-414e-aedf-be830e648883" containerID="5984429b31752f91e808ec4c27cdac411d637af501d998cac60b16cb03a39e36" exitCode=0
Oct 31 00:56:50.492649 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:50.492623    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt" event={"ID":"b046f39f-ed7f-414e-aedf-be830e648883","Type":"ContainerDied","Data":"5984429b31752f91e808ec4c27cdac411d637af501d998cac60b16cb03a39e36"}
Oct 31 00:56:50.493270 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:50.493130    4327 scope.go:122] "RemoveContainer" containerID="5984429b31752f91e808ec4c27cdac411d637af501d998cac60b16cb03a39e36"
Oct 31 00:56:50.495519 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:50.495479    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:56:50.495652 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:50.495573    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:56:50.495652 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:50.495619    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:56:50.555797 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:50.555737    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8" podStartSLOduration=10.555717938 podStartE2EDuration="10.555717938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:42.507544544 +0000 UTC m=+2262.729580089" watchObservedRunningTime="2026-10-31 00:56:50.555717938 +0000 UTC m=+2270.777753482"
Oct 31 00:56:51.504242 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:51.504207    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt" event={"ID":"b046f39f-ed7f-414e-aedf-be830e648883","Type":"ContainerStarted","Data":"c59b84d2b1f421651193a5d0b1166f9361ebb7403a662a1cb5311ab22d982e8e"}
Oct 31 00:56:52.212760 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:52.212716    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:56:52.212973 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:52.212950    4327 patch_prober.go:29] interesting pod/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 31 00:56:52.213027 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:52.213003    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt" podUID="b046f39f-ed7f-414e-aedf-be830e648883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 31 00:56:53.212657 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:53.212619    4327 patch_prober.go:29] interesting pod/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 31 00:56:53.213108 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:53.212704    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt" podUID="b046f39f-ed7f-414e-aedf-be830e648883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 31 00:56:53.423704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:53.422895    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng"
Oct 31 00:56:53.431701 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:53.429874    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8"
Oct 31 00:56:54.213439 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:54.213401    4327 patch_prober.go:29] interesting pod/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 31 00:56:54.213906 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:54.213455    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt" podUID="b046f39f-ed7f-414e-aedf-be830e648883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 31 00:56:55.247788 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:55.244818    4327 patch_prober.go:29] interesting pod/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 31 00:56:55.247788 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:55.244893    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt" podUID="b046f39f-ed7f-414e-aedf-be830e648883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 31 00:56:56.212608 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:56.212572    4327 patch_prober.go:29] interesting pod/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 31 00:56:56.212819 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:56.212628    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt" podUID="b046f39f-ed7f-414e-aedf-be830e648883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 31 00:56:57.217712 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:57.217666    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:56:57.218251 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:57.218228    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:56:57.218901 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:56:57.218880    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 00:57:15.293749 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:57:15.293625    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:57:25.148088 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:25.148054    4327 generic.go:356] "Generic (PLEG): container finished" podID="20f13ccf-97ff-4e16-ba0b-c050043ce8c8" containerID="2ace50a97b7ca77da7c480b71ce106487ced0112d72fcb42d69e640c98049cad" exitCode=0
Oct 31 00:57:25.148654 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:25.148164    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9" event={"ID":"20f13ccf-97ff-4e16-ba0b-c050043ce8c8","Type":"ContainerDied","Data":"2ace50a97b7ca77da7c480b71ce106487ced0112d72fcb42d69e640c98049cad"}
Oct 31 00:57:25.149272 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:25.149249    4327 scope.go:122] "RemoveContainer" containerID="2ace50a97b7ca77da7c480b71ce106487ced0112d72fcb42d69e640c98049cad"
Oct 31 00:57:25.151511 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:25.151474    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:57:25.151590 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:25.151557    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:57:25.151629 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:25.151594    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:57:26.158588 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:26.158552    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9" event={"ID":"20f13ccf-97ff-4e16-ba0b-c050043ce8c8","Type":"ContainerStarted","Data":"493baec8d2b60234b949a70d452589debc8a0c778ab3b7babd25782bae384e89"}
Oct 31 00:57:26.375196 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:26.375158    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:57:26.376374 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:26.376346    4327 patch_prober.go:29] interesting pod/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 31 00:57:26.376506 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:26.376394    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9" podUID="20f13ccf-97ff-4e16-ba0b-c050043ce8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 31 00:57:27.376116 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:27.376072    4327 patch_prober.go:29] interesting pod/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 31 00:57:27.376516 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:27.376161    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9" podUID="20f13ccf-97ff-4e16-ba0b-c050043ce8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 31 00:57:28.375726 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:28.375658    4327 patch_prober.go:29] interesting pod/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 31 00:57:28.375933 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:28.375753    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9" podUID="20f13ccf-97ff-4e16-ba0b-c050043ce8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 31 00:57:29.375610 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:29.375576    4327 patch_prober.go:29] interesting pod/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 31 00:57:29.376070 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:29.375634    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9" podUID="20f13ccf-97ff-4e16-ba0b-c050043ce8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 31 00:57:30.376704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:30.376276    4327 patch_prober.go:29] interesting pod/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 31 00:57:30.376704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:30.376361    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9" podUID="20f13ccf-97ff-4e16-ba0b-c050043ce8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 31 00:57:31.375839 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:31.375803    4327 patch_prober.go:29] interesting pod/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 31 00:57:31.376046 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:31.375879    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9" podUID="20f13ccf-97ff-4e16-ba0b-c050043ce8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 31 00:57:31.859845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:31.859803    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:57:32.381078 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:32.381051    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:57:32.382509 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:32.382483    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 00:57:42.897805 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:42.897770    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2"]
Oct 31 00:57:42.903034 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:42.903009    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2"
Oct 31 00:57:42.924151 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:42.924120    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2"]
Oct 31 00:57:42.934708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:42.929414    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4f4e69\""
Oct 31 00:57:42.934708 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:42.929507    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4f4e69-dockercfg-tftgp\""
Oct 31 00:57:42.950917 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:42.950877    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k8rw\" (UniqueName: \"kubernetes.io/projected/881a9204-bc36-4e4f-bb7d-89715ac273fb-kube-api-access-5k8rw\") pod \"payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2\" (UID: \"881a9204-bc36-4e4f-bb7d-89715ac273fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2"
Oct 31 00:57:42.951081 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:42.951048    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/881a9204-bc36-4e4f-bb7d-89715ac273fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2\" (UID: \"881a9204-bc36-4e4f-bb7d-89715ac273fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2"
Oct 31 00:57:43.084742 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:43.084698    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k8rw\" (UniqueName: \"kubernetes.io/projected/881a9204-bc36-4e4f-bb7d-89715ac273fb-kube-api-access-5k8rw\") pod \"payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2\" (UID: \"881a9204-bc36-4e4f-bb7d-89715ac273fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2"
Oct 31 00:57:43.103704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:43.096155    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/881a9204-bc36-4e4f-bb7d-89715ac273fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2\" (UID: \"881a9204-bc36-4e4f-bb7d-89715ac273fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2"
Oct 31 00:57:43.103704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:43.097547    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/881a9204-bc36-4e4f-bb7d-89715ac273fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2\" (UID: \"881a9204-bc36-4e4f-bb7d-89715ac273fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2"
Oct 31 00:57:43.152546 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:43.152466    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k8rw\" (UniqueName: \"kubernetes.io/projected/881a9204-bc36-4e4f-bb7d-89715ac273fb-kube-api-access-5k8rw\") pod \"payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2\" (UID: \"881a9204-bc36-4e4f-bb7d-89715ac273fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2"
Oct 31 00:57:43.304700 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:43.287226    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2"
Oct 31 00:57:43.684859 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:43.684827    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2"]
Oct 31 00:57:43.685206 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:57:43.685178    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod881a9204_bc36_4e4f_bb7d_89715ac273fb.slice/crio-7fd62b1eef9fe7b40d1d9df6f38875ed138493d1c7f913ed307419e57e98ccbe WatchSource:0}: Error finding container 7fd62b1eef9fe7b40d1d9df6f38875ed138493d1c7f913ed307419e57e98ccbe: Status 404 returned error can't find the container with id 7fd62b1eef9fe7b40d1d9df6f38875ed138493d1c7f913ed307419e57e98ccbe
Oct 31 00:57:44.385074 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:44.385037    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2" event={"ID":"881a9204-bc36-4e4f-bb7d-89715ac273fb","Type":"ContainerStarted","Data":"b4f2b4f1110172858e52446b1d97c8ccade36f41c63cf1710f0a12c3e6401b59"}
Oct 31 00:57:44.385485 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:44.385089    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2" event={"ID":"881a9204-bc36-4e4f-bb7d-89715ac273fb","Type":"ContainerStarted","Data":"7fd62b1eef9fe7b40d1d9df6f38875ed138493d1c7f913ed307419e57e98ccbe"}
Oct 31 00:57:44.385485 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:44.385295    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2"
Oct 31 00:57:44.429276 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:44.429209    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2" podStartSLOduration=2.429189132 podStartE2EDuration="2.429189132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:44.426821699 +0000 UTC m=+2324.648857241" watchObservedRunningTime="2026-10-31 00:57:44.429189132 +0000 UTC m=+2324.651224678"
Oct 31 00:57:45.294452 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:57:45.294411    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:57:55.395605 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:57:55.395572    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2"
Oct 31 00:58:09.912844 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:09.912797    4327 patch_prober.go:29] interesting pod/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:58:09.913318 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:09.912897    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz" podUID="9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:58:12.705273 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:12.705185    4327 generic.go:356] "Generic (PLEG): container finished" podID="9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3" containerID="36af665e2edb2165b20f033f90248a1eb21445866c3f379f14a0d09c59861dcc" exitCode=0
Oct 31 00:58:12.705841 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:12.705262    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz" event={"ID":"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3","Type":"ContainerDied","Data":"36af665e2edb2165b20f033f90248a1eb21445866c3f379f14a0d09c59861dcc"}
Oct 31 00:58:12.705913 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:12.705884    4327 scope.go:122] "RemoveContainer" containerID="36af665e2edb2165b20f033f90248a1eb21445866c3f379f14a0d09c59861dcc"
Oct 31 00:58:12.707906 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:12.707870    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:58:12.708025 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:12.707963    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:58:12.708025 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:12.708003    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 00:58:13.715931 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:13.715887    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz" event={"ID":"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3","Type":"ContainerStarted","Data":"643e90a2b6044dcd604705694292c07f068e0758c6f11dc689b1e4ef6d9a8604"}
Oct 31 00:58:14.540134 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:14.540098    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:58:14.540495 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:14.540470    4327 patch_prober.go:29] interesting pod/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 31 00:58:14.540699 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:14.540652    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz" podUID="9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 31 00:58:15.296184 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:58:15.296139    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:58:15.538718 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:15.538668    4327 patch_prober.go:29] interesting pod/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 31 00:58:15.539026 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:15.538998    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz" podUID="9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 31 00:58:16.538122 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:16.538082    4327 patch_prober.go:29] interesting pod/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 31 00:58:16.538746 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:16.538714    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz" podUID="9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 31 00:58:17.193041 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.193007    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"]
Oct 31 00:58:17.202049 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.202022    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"
Oct 31 00:58:17.226572 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.226539    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-72120c\"/\"openshift-service-ca.crt\""
Oct 31 00:58:17.253714 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.249616    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-72120c\"/\"kube-root-ca.crt\""
Oct 31 00:58:17.260079 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.257429    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-72120c\"/\"default-dockercfg-r5hhc\""
Oct 31 00:58:17.260315 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.257514    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-72120c\"/\"test-model-72120c-kserve-self-signed-certs\""
Oct 31 00:58:17.263616 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.263592    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"]
Oct 31 00:58:17.356238 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.356205    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-model-cache\") pod \"test-model-72120c-kserve-7696d9f9d5-d2sh5\" (UID: \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\") " pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"
Oct 31 00:58:17.356545 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.356520    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-dshm\") pod \"test-model-72120c-kserve-7696d9f9d5-d2sh5\" (UID: \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\") " pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"
Oct 31 00:58:17.356671 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.356654    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-tmp-dir\") pod \"test-model-72120c-kserve-7696d9f9d5-d2sh5\" (UID: \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\") " pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"
Oct 31 00:58:17.357362 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.357340    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-home\") pod \"test-model-72120c-kserve-7696d9f9d5-d2sh5\" (UID: \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\") " pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"
Oct 31 00:58:17.357538 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.357520    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv8wd\" (UniqueName: \"kubernetes.io/projected/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-kube-api-access-zv8wd\") pod \"test-model-72120c-kserve-7696d9f9d5-d2sh5\" (UID: \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\") " pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"
Oct 31 00:58:17.357856 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.357836    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-tls-certs\") pod \"test-model-72120c-kserve-7696d9f9d5-d2sh5\" (UID: \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\") " pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"
Oct 31 00:58:17.460291 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.460194    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-model-cache\") pod \"test-model-72120c-kserve-7696d9f9d5-d2sh5\" (UID: \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\") " pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"
Oct 31 00:58:17.460471 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.460312    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-dshm\") pod \"test-model-72120c-kserve-7696d9f9d5-d2sh5\" (UID: \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\") " pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"
Oct 31 00:58:17.460471 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.460355    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-tmp-dir\") pod \"test-model-72120c-kserve-7696d9f9d5-d2sh5\" (UID: \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\") " pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"
Oct 31 00:58:17.460471 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.460387    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-home\") pod \"test-model-72120c-kserve-7696d9f9d5-d2sh5\" (UID: \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\") " pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"
Oct 31 00:58:17.460471 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.460464    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv8wd\" (UniqueName: \"kubernetes.io/projected/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-kube-api-access-zv8wd\") pod \"test-model-72120c-kserve-7696d9f9d5-d2sh5\" (UID: \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\") " pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"
Oct 31 00:58:17.460727 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.460708    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-tls-certs\") pod \"test-model-72120c-kserve-7696d9f9d5-d2sh5\" (UID: \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\") " pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"
Oct 31 00:58:17.488705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.485193    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-tmp-dir\") pod \"test-model-72120c-kserve-7696d9f9d5-d2sh5\" (UID: \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\") " pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"
Oct 31 00:58:17.488705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.485452    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-model-cache\") pod \"test-model-72120c-kserve-7696d9f9d5-d2sh5\" (UID: \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\") " pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"
Oct 31 00:58:17.488705 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.486044    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-home\") pod \"test-model-72120c-kserve-7696d9f9d5-d2sh5\" (UID: \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\") " pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"
Oct 31 00:58:17.502202 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.502167    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-dshm\") pod \"test-model-72120c-kserve-7696d9f9d5-d2sh5\" (UID: \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\") " pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"
Oct 31 00:58:17.533540 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.533499    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-tls-certs\") pod \"test-model-72120c-kserve-7696d9f9d5-d2sh5\" (UID: \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\") " pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"
Oct 31 00:58:17.536424 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.536387    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv8wd\" (UniqueName: \"kubernetes.io/projected/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-kube-api-access-zv8wd\") pod \"test-model-72120c-kserve-7696d9f9d5-d2sh5\" (UID: \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\") " pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"
Oct 31 00:58:17.546697 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.539775    4327 patch_prober.go:29] interesting pod/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 31 00:58:17.546697 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.539841    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz" podUID="9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 31 00:58:17.566724 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:17.565302    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"
Oct 31 00:58:18.047117 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:58:18.047058    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3b9b681_9675_4c15_81d4_0fed4b4b2d63.slice/crio-f95c26e1006b74ff88e46089b8dedec009323effc050a6ce87680b6ecfb32af2 WatchSource:0}: Error finding container f95c26e1006b74ff88e46089b8dedec009323effc050a6ce87680b6ecfb32af2: Status 404 returned error can't find the container with id f95c26e1006b74ff88e46089b8dedec009323effc050a6ce87680b6ecfb32af2
Oct 31 00:58:18.047566 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:18.047526    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"]
Oct 31 00:58:18.541704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:18.541079    4327 patch_prober.go:29] interesting pod/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 31 00:58:18.541704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:18.541142    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz" podUID="9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 31 00:58:18.807447 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:18.807332    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5" event={"ID":"c3b9b681-9675-4c15-81d4-0fed4b4b2d63","Type":"ContainerStarted","Data":"ae457bec7ea24817b44c755840ac69747682150e980b41e62f8d124783257a01"}
Oct 31 00:58:18.807447 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:18.807399    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5" event={"ID":"c3b9b681-9675-4c15-81d4-0fed4b4b2d63","Type":"ContainerStarted","Data":"f95c26e1006b74ff88e46089b8dedec009323effc050a6ce87680b6ecfb32af2"}
Oct 31 00:58:18.875993 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:18.875932    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5" podStartSLOduration=1.8759151429999998 podStartE2EDuration="1.875915143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:58:18.870497089 +0000 UTC m=+2359.092532631" watchObservedRunningTime="2026-10-31 00:58:18.875915143 +0000 UTC m=+2359.097950687"
Oct 31 00:58:19.554568 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:19.554531    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:58:19.555158 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:19.555129    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:58:19.556415 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:19.556388    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 00:58:19.567595 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:19.567573    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"
Oct 31 00:58:27.567256 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:27.567207    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"
Oct 31 00:58:27.579978 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:27.579949    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"
Oct 31 00:58:27.895740 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:27.895711    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"
Oct 31 00:58:45.296154 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:58:45.296111    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:58:52.521999 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:52.521960    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"]
Oct 31 00:58:52.527501 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:52.527474    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"
Oct 31 00:58:52.550028 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:52.549997    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"]
Oct 31 00:58:52.557372 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:52.557341    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-82b5e8\"/\"kube-root-ca.crt\""
Oct 31 00:58:52.558517 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:52.557443    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-82b5e8\"/\"openshift-service-ca.crt\""
Oct 31 00:58:52.558517 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:52.557564    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-82b5e8\"/\"test-model-82b5e8-kserve-self-signed-certs\""
Oct 31 00:58:52.558702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:52.558519    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-82b5e8\"/\"default-dockercfg-6rllf\""
Oct 31 00:58:52.661584 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:52.661542    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-model-cache\") pod \"test-model-82b5e8-kserve-5cb569446c-7xngz\" (UID: \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\") " pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"
Oct 31 00:58:52.661584 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:52.661585    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6987v\" (UniqueName: \"kubernetes.io/projected/e0a4bc42-fcaa-47bc-ad35-a654443342c1-kube-api-access-6987v\") pod \"test-model-82b5e8-kserve-5cb569446c-7xngz\" (UID: \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\") " pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"
Oct 31 00:58:52.661895 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:52.661875    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-home\") pod \"test-model-82b5e8-kserve-5cb569446c-7xngz\" (UID: \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\") " pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"
Oct 31 00:58:52.662063 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:52.662041    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-tmp-dir\") pod \"test-model-82b5e8-kserve-5cb569446c-7xngz\" (UID: \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\") " pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"
Oct 31 00:58:52.662114 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:52.662083    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0a4bc42-fcaa-47bc-ad35-a654443342c1-tls-certs\") pod \"test-model-82b5e8-kserve-5cb569446c-7xngz\" (UID: \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\") " pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"
Oct 31 00:58:52.662219 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:52.662199    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-dshm\") pod \"test-model-82b5e8-kserve-5cb569446c-7xngz\" (UID: \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\") " pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"
Oct 31 00:58:52.764294 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:52.764259    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-tmp-dir\") pod \"test-model-82b5e8-kserve-5cb569446c-7xngz\" (UID: \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\") " pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"
Oct 31 00:58:52.764476 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:52.764308    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0a4bc42-fcaa-47bc-ad35-a654443342c1-tls-certs\") pod \"test-model-82b5e8-kserve-5cb569446c-7xngz\" (UID: \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\") " pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"
Oct 31 00:58:52.764476 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:52.764372    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-dshm\") pod \"test-model-82b5e8-kserve-5cb569446c-7xngz\" (UID: \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\") " pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"
Oct 31 00:58:52.764476 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:52.764450    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-model-cache\") pod \"test-model-82b5e8-kserve-5cb569446c-7xngz\" (UID: \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\") " pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"
Oct 31 00:58:52.764648 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:52.764479    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6987v\" (UniqueName: \"kubernetes.io/projected/e0a4bc42-fcaa-47bc-ad35-a654443342c1-kube-api-access-6987v\") pod \"test-model-82b5e8-kserve-5cb569446c-7xngz\" (UID: \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\") " pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"
Oct 31 00:58:52.764725 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:52.764699    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-home\") pod \"test-model-82b5e8-kserve-5cb569446c-7xngz\" (UID: \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\") " pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"
Oct 31 00:58:52.764792 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:52.764727    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-tmp-dir\") pod \"test-model-82b5e8-kserve-5cb569446c-7xngz\" (UID: \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\") " pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"
Oct 31 00:58:52.764926 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:52.764902    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-model-cache\") pod \"test-model-82b5e8-kserve-5cb569446c-7xngz\" (UID: \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\") " pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"
Oct 31 00:58:52.764992 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:52.764982    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-home\") pod \"test-model-82b5e8-kserve-5cb569446c-7xngz\" (UID: \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\") " pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"
Oct 31 00:58:52.766822 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:52.766800    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-dshm\") pod \"test-model-82b5e8-kserve-5cb569446c-7xngz\" (UID: \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\") " pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"
Oct 31 00:58:52.767199 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:52.767176    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0a4bc42-fcaa-47bc-ad35-a654443342c1-tls-certs\") pod \"test-model-82b5e8-kserve-5cb569446c-7xngz\" (UID: \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\") " pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"
Oct 31 00:58:52.800744 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:52.800611    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6987v\" (UniqueName: \"kubernetes.io/projected/e0a4bc42-fcaa-47bc-ad35-a654443342c1-kube-api-access-6987v\") pod \"test-model-82b5e8-kserve-5cb569446c-7xngz\" (UID: \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\") " pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"
Oct 31 00:58:52.843033 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:52.842994    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"
Oct 31 00:58:53.314071 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:53.314043    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"]
Oct 31 00:58:53.314984 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:58:53.314955    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0a4bc42_fcaa_47bc_ad35_a654443342c1.slice/crio-523adf7b00bb2d2a98c637ff5e2f885c69c852181019849db9b9576cc4a4f449 WatchSource:0}: Error finding container 523adf7b00bb2d2a98c637ff5e2f885c69c852181019849db9b9576cc4a4f449: Status 404 returned error can't find the container with id 523adf7b00bb2d2a98c637ff5e2f885c69c852181019849db9b9576cc4a4f449
Oct 31 00:58:54.133454 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:54.133412    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz" event={"ID":"e0a4bc42-fcaa-47bc-ad35-a654443342c1","Type":"ContainerStarted","Data":"8577448987a5d663f5c93b2cc8bce055bba2a79f29a9cc91d8da476225bb6012"}
Oct 31 00:58:54.133966 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:54.133470    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz" event={"ID":"e0a4bc42-fcaa-47bc-ad35-a654443342c1","Type":"ContainerStarted","Data":"523adf7b00bb2d2a98c637ff5e2f885c69c852181019849db9b9576cc4a4f449"}
Oct 31 00:58:54.179325 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:54.179269    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz" podStartSLOduration=2.179254813 podStartE2EDuration="2.179254813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:58:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:58:54.176331833 +0000 UTC m=+2394.398367375" watchObservedRunningTime="2026-10-31 00:58:54.179254813 +0000 UTC m=+2394.401290354"
Oct 31 00:58:54.848674 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:54.848637    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"
Oct 31 00:58:56.605323 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:56.605282    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"]
Oct 31 00:58:56.606161 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:58:56.606124    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp" podUID="004c3b2b-bdd7-42df-a961-3ce60631a8b4" containerName="istio-proxy" containerID="cri-o://fd9b6b017c1ec9da5fa9f6a07a015f2f0f0a1291610550f0d516d1708bf8a7e2" gracePeriod=30
Oct 31 00:59:01.464084 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:01.464052    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:59:01.464461 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:01.464114    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp" podUID="004c3b2b-bdd7-42df-a961-3ce60631a8b4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:59:01.920405 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:01.920375    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:59:02.058010 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.057904    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-workload-socket\") pod \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") "
Oct 31 00:59:02.058010 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.057980    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istiod-ca-cert\") pod \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") "
Oct 31 00:59:02.058257 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.058044    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-workload-certs\") pod \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") "
Oct 31 00:59:02.058257 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.058111    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-envoy\") pod \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") "
Oct 31 00:59:02.058257 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.058204    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-credential-socket\") pod \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") "
Oct 31 00:59:02.058411 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.058234    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-workload-socket" pod "004c3b2b-bdd7-42df-a961-3ce60631a8b4" (UID: "004c3b2b-bdd7-42df-a961-3ce60631a8b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:59:02.058411 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.058306    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-token\" (UniqueName: \"kubernetes.io/projected/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-token\") pod \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") "
Oct 31 00:59:02.058525 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.058398    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-workload-certs" pod "004c3b2b-bdd7-42df-a961-3ce60631a8b4" (UID: "004c3b2b-bdd7-42df-a961-3ce60631a8b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:59:02.058525 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.058401    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istiod-ca-cert" pod "004c3b2b-bdd7-42df-a961-3ce60631a8b4" (UID: "004c3b2b-bdd7-42df-a961-3ce60631a8b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:59:02.058525 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.058486    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-credential-socket" pod "004c3b2b-bdd7-42df-a961-3ce60631a8b4" (UID: "004c3b2b-bdd7-42df-a961-3ce60631a8b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:59:02.058729 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.058624    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/004c3b2b-bdd7-42df-a961-3ce60631a8b4-kube-api-access-zm7km\" (UniqueName: \"kubernetes.io/projected/004c3b2b-bdd7-42df-a961-3ce60631a8b4-kube-api-access-zm7km\") pod \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") "
Oct 31 00:59:02.058729 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.058649    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-data\") pod \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") "
Oct 31 00:59:02.058729 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.058669    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-podinfo\") pod \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\" (UID: \"004c3b2b-bdd7-42df-a961-3ce60631a8b4\") "
Oct 31 00:59:02.058992 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.058953    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-data" pod "004c3b2b-bdd7-42df-a961-3ce60631a8b4" (UID: "004c3b2b-bdd7-42df-a961-3ce60631a8b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:59:02.060234 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.060105    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:59:02.060234 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.060155    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:59:02.060234 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.060172    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:59:02.060234 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.060187    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:59:02.060234 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.060201    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:59:02.060996 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.060967    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-token" pod "004c3b2b-bdd7-42df-a961-3ce60631a8b4" (UID: "004c3b2b-bdd7-42df-a961-3ce60631a8b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:59:02.061142 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.061122    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004c3b2b-bdd7-42df-a961-3ce60631a8b4-kube-api-access-zm7km" pod "004c3b2b-bdd7-42df-a961-3ce60631a8b4" (UID: "004c3b2b-bdd7-42df-a961-3ce60631a8b4"). InnerVolumeSpecName "kube-api-access-zm7km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:59:02.061202 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.061162    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-envoy" pod "004c3b2b-bdd7-42df-a961-3ce60631a8b4" (UID: "004c3b2b-bdd7-42df-a961-3ce60631a8b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:59:02.061456 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.061440    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-podinfo" pod "004c3b2b-bdd7-42df-a961-3ce60631a8b4" (UID: "004c3b2b-bdd7-42df-a961-3ce60631a8b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:59:02.162178 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.162139    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:59:02.162178 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.162171    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:59:02.162178 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.162181    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm7km\" (UniqueName: \"kubernetes.io/projected/004c3b2b-bdd7-42df-a961-3ce60631a8b4-kube-api-access-zm7km\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:59:02.162420 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.162190    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004c3b2b-bdd7-42df-a961-3ce60631a8b4-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:59:02.291246 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.291210    4327 generic.go:356] "Generic (PLEG): container finished" podID="004c3b2b-bdd7-42df-a961-3ce60631a8b4" containerID="fd9b6b017c1ec9da5fa9f6a07a015f2f0f0a1291610550f0d516d1708bf8a7e2" exitCode=0
Oct 31 00:59:02.291444 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.291254    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"
Oct 31 00:59:02.291444 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.291290    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp" event={"ID":"004c3b2b-bdd7-42df-a961-3ce60631a8b4","Type":"ContainerDied","Data":"fd9b6b017c1ec9da5fa9f6a07a015f2f0f0a1291610550f0d516d1708bf8a7e2"}
Oct 31 00:59:02.291444 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.291350    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp" event={"ID":"004c3b2b-bdd7-42df-a961-3ce60631a8b4","Type":"ContainerDied","Data":"e1336d0e589fabd0eb8bb4a1aa3a03fcd0ff21e85282d1ff4806c1407775898e"}
Oct 31 00:59:02.291444 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.291376    4327 scope.go:122] "RemoveContainer" containerID="fd9b6b017c1ec9da5fa9f6a07a015f2f0f0a1291610550f0d516d1708bf8a7e2"
Oct 31 00:59:02.304610 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.304592    4327 scope.go:122] "RemoveContainer" containerID="c1c99c775dd03e98676e576977294ef0847c4e7acccbb1df88db67cf093a330e"
Oct 31 00:59:02.317335 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.317307    4327 scope.go:122] "RemoveContainer" containerID="fd9b6b017c1ec9da5fa9f6a07a015f2f0f0a1291610550f0d516d1708bf8a7e2"
Oct 31 00:59:02.317772 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:59:02.317743    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd9b6b017c1ec9da5fa9f6a07a015f2f0f0a1291610550f0d516d1708bf8a7e2\": container with ID starting with fd9b6b017c1ec9da5fa9f6a07a015f2f0f0a1291610550f0d516d1708bf8a7e2 not found: ID does not exist" containerID="fd9b6b017c1ec9da5fa9f6a07a015f2f0f0a1291610550f0d516d1708bf8a7e2"
Oct 31 00:59:02.317883 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.317780    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9b6b017c1ec9da5fa9f6a07a015f2f0f0a1291610550f0d516d1708bf8a7e2"} err="failed to get container status \"fd9b6b017c1ec9da5fa9f6a07a015f2f0f0a1291610550f0d516d1708bf8a7e2\": rpc error: code = NotFound desc = could not find container \"fd9b6b017c1ec9da5fa9f6a07a015f2f0f0a1291610550f0d516d1708bf8a7e2\": container with ID starting with fd9b6b017c1ec9da5fa9f6a07a015f2f0f0a1291610550f0d516d1708bf8a7e2 not found: ID does not exist"
Oct 31 00:59:02.317883 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.317799    4327 scope.go:122] "RemoveContainer" containerID="c1c99c775dd03e98676e576977294ef0847c4e7acccbb1df88db67cf093a330e"
Oct 31 00:59:02.318104 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:59:02.318074    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c99c775dd03e98676e576977294ef0847c4e7acccbb1df88db67cf093a330e\": container with ID starting with c1c99c775dd03e98676e576977294ef0847c4e7acccbb1df88db67cf093a330e not found: ID does not exist" containerID="c1c99c775dd03e98676e576977294ef0847c4e7acccbb1df88db67cf093a330e"
Oct 31 00:59:02.318154 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.318115    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c99c775dd03e98676e576977294ef0847c4e7acccbb1df88db67cf093a330e"} err="failed to get container status \"c1c99c775dd03e98676e576977294ef0847c4e7acccbb1df88db67cf093a330e\": rpc error: code = NotFound desc = could not find container \"c1c99c775dd03e98676e576977294ef0847c4e7acccbb1df88db67cf093a330e\": container with ID starting with c1c99c775dd03e98676e576977294ef0847c4e7acccbb1df88db67cf093a330e not found: ID does not exist"
Oct 31 00:59:02.357187 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.357151    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"]
Oct 31 00:59:02.367171 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.367142    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-b4qbp"]
Oct 31 00:59:02.848885 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.848846    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"
Oct 31 00:59:02.862754 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.862717    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"
Oct 31 00:59:02.871428 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.871398    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt"]
Oct 31 00:59:02.872378 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.872358    4327 cpu_manager.go:414] "containerMap: removing container" podUID="004c3b2b-bdd7-42df-a961-3ce60631a8b4" containerName="istio-proxy"
Oct 31 00:59:02.872378 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.872379    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="004c3b2b-bdd7-42df-a961-3ce60631a8b4" containerName="istio-proxy"
Oct 31 00:59:02.872530 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.872513    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="004c3b2b-bdd7-42df-a961-3ce60631a8b4" containerName="istio-proxy"
Oct 31 00:59:02.872530 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.872526    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="004c3b2b-bdd7-42df-a961-3ce60631a8b4" containerName="istio-proxy"
Oct 31 00:59:02.890295 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.890264    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt"
Oct 31 00:59:02.901826 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.901793    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt"]
Oct 31 00:59:02.907629 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.907606    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-6f8856\""
Oct 31 00:59:02.908026 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:02.907963    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-6f8856-dockercfg-66pdj\""
Oct 31 00:59:03.007711 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:03.000943    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ee1484c-003a-4b17-af4a-830ca58a1691-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt\" (UID: \"4ee1484c-003a-4b17-af4a-830ca58a1691\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt"
Oct 31 00:59:03.007711 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:03.001212    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vvwn\" (UniqueName: \"kubernetes.io/projected/4ee1484c-003a-4b17-af4a-830ca58a1691-kube-api-access-5vvwn\") pod \"payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt\" (UID: \"4ee1484c-003a-4b17-af4a-830ca58a1691\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt"
Oct 31 00:59:03.045702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:03.043774    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl"]
Oct 31 00:59:03.045702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:03.045148    4327 cpu_manager.go:414] "containerMap: removing container" podUID="004c3b2b-bdd7-42df-a961-3ce60631a8b4" containerName="istio-proxy"
Oct 31 00:59:03.045702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:03.045174    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="004c3b2b-bdd7-42df-a961-3ce60631a8b4" containerName="istio-proxy"
Oct 31 00:59:03.116770 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:03.108208    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ee1484c-003a-4b17-af4a-830ca58a1691-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt\" (UID: \"4ee1484c-003a-4b17-af4a-830ca58a1691\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt"
Oct 31 00:59:03.116770 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:03.108377    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vvwn\" (UniqueName: \"kubernetes.io/projected/4ee1484c-003a-4b17-af4a-830ca58a1691-kube-api-access-5vvwn\") pod \"payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt\" (UID: \"4ee1484c-003a-4b17-af4a-830ca58a1691\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt"
Oct 31 00:59:03.116770 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:03.112349    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ee1484c-003a-4b17-af4a-830ca58a1691-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt\" (UID: \"4ee1484c-003a-4b17-af4a-830ca58a1691\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt"
Oct 31 00:59:03.140707 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:03.124996    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl"
Oct 31 00:59:03.155291 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:03.142742    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl"]
Oct 31 00:59:03.155291 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:03.146756    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vvwn\" (UniqueName: \"kubernetes.io/projected/4ee1484c-003a-4b17-af4a-830ca58a1691-kube-api-access-5vvwn\") pod \"payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt\" (UID: \"4ee1484c-003a-4b17-af4a-830ca58a1691\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt"
Oct 31 00:59:03.217753 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:03.214958    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt"
Oct 31 00:59:03.316031 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:03.314255    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc06dda6-d818-4a58-89d3-8eb40043b464-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl\" (UID: \"cc06dda6-d818-4a58-89d3-8eb40043b464\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl"
Oct 31 00:59:03.316031 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:03.314423    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4cf6\" (UniqueName: \"kubernetes.io/projected/cc06dda6-d818-4a58-89d3-8eb40043b464-kube-api-access-r4cf6\") pod \"payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl\" (UID: \"cc06dda6-d818-4a58-89d3-8eb40043b464\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl"
Oct 31 00:59:03.318338 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:03.318314    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"
Oct 31 00:59:03.418775 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:03.416157    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc06dda6-d818-4a58-89d3-8eb40043b464-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl\" (UID: \"cc06dda6-d818-4a58-89d3-8eb40043b464\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl"
Oct 31 00:59:03.418775 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:03.416389    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4cf6\" (UniqueName: \"kubernetes.io/projected/cc06dda6-d818-4a58-89d3-8eb40043b464-kube-api-access-r4cf6\") pod \"payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl\" (UID: \"cc06dda6-d818-4a58-89d3-8eb40043b464\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl"
Oct 31 00:59:03.418775 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:03.417084    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc06dda6-d818-4a58-89d3-8eb40043b464-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl\" (UID: \"cc06dda6-d818-4a58-89d3-8eb40043b464\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl"
Oct 31 00:59:03.443234 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:03.443202    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4cf6\" (UniqueName: \"kubernetes.io/projected/cc06dda6-d818-4a58-89d3-8eb40043b464-kube-api-access-r4cf6\") pod \"payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl\" (UID: \"cc06dda6-d818-4a58-89d3-8eb40043b464\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl"
Oct 31 00:59:03.567505 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:03.567476    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt"]
Oct 31 00:59:03.567624 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:59:03.567594    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee1484c_003a_4b17_af4a_830ca58a1691.slice/crio-086ef5af30b9bb6c6aa3ee95d2224e8e7e733b6b5d1d2b825ad6f541403cb3ac WatchSource:0}: Error finding container 086ef5af30b9bb6c6aa3ee95d2224e8e7e733b6b5d1d2b825ad6f541403cb3ac: Status 404 returned error can't find the container with id 086ef5af30b9bb6c6aa3ee95d2224e8e7e733b6b5d1d2b825ad6f541403cb3ac
Oct 31 00:59:03.589424 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:03.589392    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl"
Oct 31 00:59:03.909568 ip-10-0-128-253 kubenswrapper[4327]: W1031 00:59:03.909526    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc06dda6_d818_4a58_89d3_8eb40043b464.slice/crio-ea95a7a1ae759f4c376668571e807cedfef8ce7a3fd48b0ef303e38b7f56dc4c WatchSource:0}: Error finding container ea95a7a1ae759f4c376668571e807cedfef8ce7a3fd48b0ef303e38b7f56dc4c: Status 404 returned error can't find the container with id ea95a7a1ae759f4c376668571e807cedfef8ce7a3fd48b0ef303e38b7f56dc4c
Oct 31 00:59:03.909966 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:03.909921    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl"]
Oct 31 00:59:03.925433 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:03.925372    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="004c3b2b-bdd7-42df-a961-3ce60631a8b4" path="/var/lib/kubelet/pods/004c3b2b-bdd7-42df-a961-3ce60631a8b4/volumes"
Oct 31 00:59:04.371558 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:04.370502    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl" event={"ID":"cc06dda6-d818-4a58-89d3-8eb40043b464","Type":"ContainerStarted","Data":"1ef143008ef5a7d320334b1bb9ebb9298d92e04ae041bdbf83dc2a8acd96f339"}
Oct 31 00:59:04.371558 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:04.370833    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl" event={"ID":"cc06dda6-d818-4a58-89d3-8eb40043b464","Type":"ContainerStarted","Data":"ea95a7a1ae759f4c376668571e807cedfef8ce7a3fd48b0ef303e38b7f56dc4c"}
Oct 31 00:59:04.371558 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:04.370916    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl"
Oct 31 00:59:04.373195 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:04.373166    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt" event={"ID":"4ee1484c-003a-4b17-af4a-830ca58a1691","Type":"ContainerStarted","Data":"4d3506607085a398122bbc0dac327bdb6d3a467b6888eaf359ffb904282be4ba"}
Oct 31 00:59:04.373349 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:04.373224    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt" event={"ID":"4ee1484c-003a-4b17-af4a-830ca58a1691","Type":"ContainerStarted","Data":"086ef5af30b9bb6c6aa3ee95d2224e8e7e733b6b5d1d2b825ad6f541403cb3ac"}
Oct 31 00:59:04.373553 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:04.373538    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt"
Oct 31 00:59:04.418615 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:04.418547    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl" podStartSLOduration=2.418529383 podStartE2EDuration="2.418529383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:59:04.415838639 +0000 UTC m=+2404.637874181" watchObservedRunningTime="2026-10-31 00:59:04.418529383 +0000 UTC m=+2404.640564926"
Oct 31 00:59:04.459309 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:04.459251    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt" podStartSLOduration=2.459236307 podStartE2EDuration="2.459236307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:59:04.456644768 +0000 UTC m=+2404.678680310" watchObservedRunningTime="2026-10-31 00:59:04.459236307 +0000 UTC m=+2404.681271865"
Oct 31 00:59:15.294177 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:59:15.294132    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:59:15.392523 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:15.392491    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt"
Oct 31 00:59:15.392975 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:15.392946    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl"
Oct 31 00:59:44.434481 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:44.434450    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"]
Oct 31 00:59:45.297400 ip-10-0-128-253 kubenswrapper[4327]: E1031 00:59:45.297361    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 00:59:46.263726 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:46.260132    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z"]
Oct 31 00:59:46.283788 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:46.281627    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z"
Oct 31 00:59:46.322702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:46.318486    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-6f8856-serving-cert\""
Oct 31 00:59:46.322702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:46.319286    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z"]
Oct 31 00:59:46.397713 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:46.396639    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snrcn\" (UniqueName: \"kubernetes.io/projected/8c9dee08-2aa4-4c88-bea7-385a01d519bd-kube-api-access-snrcn\") pod \"maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z\" (UID: \"8c9dee08-2aa4-4c88-bea7-385a01d519bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z"
Oct 31 00:59:46.397713 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:46.396896    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c9dee08-2aa4-4c88-bea7-385a01d519bd-metrics-certs\") pod \"maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z\" (UID: \"8c9dee08-2aa4-4c88-bea7-385a01d519bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z"
Oct 31 00:59:46.397713 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:46.397008    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c9dee08-2aa4-4c88-bea7-385a01d519bd-maas-api-tls\") pod \"maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z\" (UID: \"8c9dee08-2aa4-4c88-bea7-385a01d519bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z"
Oct 31 00:59:46.397713 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:46.397050    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c9dee08-2aa4-4c88-bea7-385a01d519bd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z\" (UID: \"8c9dee08-2aa4-4c88-bea7-385a01d519bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z"
Oct 31 00:59:46.517576 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:46.515864    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snrcn\" (UniqueName: \"kubernetes.io/projected/8c9dee08-2aa4-4c88-bea7-385a01d519bd-kube-api-access-snrcn\") pod \"maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z\" (UID: \"8c9dee08-2aa4-4c88-bea7-385a01d519bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z"
Oct 31 00:59:46.517576 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:46.515979    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c9dee08-2aa4-4c88-bea7-385a01d519bd-metrics-certs\") pod \"maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z\" (UID: \"8c9dee08-2aa4-4c88-bea7-385a01d519bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z"
Oct 31 00:59:46.517845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:46.517769    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c9dee08-2aa4-4c88-bea7-385a01d519bd-maas-api-tls\") pod \"maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z\" (UID: \"8c9dee08-2aa4-4c88-bea7-385a01d519bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z"
Oct 31 00:59:46.517845 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:46.517814    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c9dee08-2aa4-4c88-bea7-385a01d519bd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z\" (UID: \"8c9dee08-2aa4-4c88-bea7-385a01d519bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z"
Oct 31 00:59:46.519790 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:46.518775    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c9dee08-2aa4-4c88-bea7-385a01d519bd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z\" (UID: \"8c9dee08-2aa4-4c88-bea7-385a01d519bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z"
Oct 31 00:59:46.552892 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:46.546485    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snrcn\" (UniqueName: \"kubernetes.io/projected/8c9dee08-2aa4-4c88-bea7-385a01d519bd-kube-api-access-snrcn\") pod \"maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z\" (UID: \"8c9dee08-2aa4-4c88-bea7-385a01d519bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z"
Oct 31 00:59:46.556721 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:46.553123    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c9dee08-2aa4-4c88-bea7-385a01d519bd-maas-api-tls\") pod \"maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z\" (UID: \"8c9dee08-2aa4-4c88-bea7-385a01d519bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z"
Oct 31 00:59:46.570702 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:46.557798    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c9dee08-2aa4-4c88-bea7-385a01d519bd-metrics-certs\") pod \"maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z\" (UID: \"8c9dee08-2aa4-4c88-bea7-385a01d519bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z"
Oct 31 00:59:46.778458 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:46.778374    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z"
Oct 31 00:59:48.053202 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:48.053171    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z"]
Oct 31 00:59:48.054065 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:48.054038    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z" event={"ID":"8c9dee08-2aa4-4c88-bea7-385a01d519bd","Type":"ContainerStarted","Data":"9cf65a09f98c85634adb335ecfcf141772e8e71ca11790205c8a4121334aa63e"}
Oct 31 00:59:49.058205 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:49.058166    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z" event={"ID":"8c9dee08-2aa4-4c88-bea7-385a01d519bd","Type":"ContainerStarted","Data":"47f2d0934c937efe9e4f16e32e72aecdc84f1840aa5a5ee1f662a3cc80c95884"}
Oct 31 00:59:49.058619 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:49.058238    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z"
Oct 31 00:59:49.105274 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:49.105223    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z" podStartSLOduration=2.639391737 podStartE2EDuration="3.105207826s" totalImagesPullingTime="465.816089ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:59:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:59:49.101960043 +0000 UTC m=+2449.323995588" watchObservedRunningTime="2026-10-31 00:59:49.105207826 +0000 UTC m=+2449.327243368"
Oct 31 00:59:55.074695 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:55.074654    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z"
Oct 31 00:59:59.459829 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:59.459783    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6" podUID="1b410769-3505-4588-9255-fcca23401c3e" containerName="main" containerID="cri-o://4e38930e5031186c751a3ddcba7d08966fd73b83238618c70e2d6acd0e856404" gracePeriod=45
Oct 31 00:59:59.491670 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:59.491637    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6" podUID="1b410769-3505-4588-9255-fcca23401c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.148:8000/ready\": dial tcp 10.133.0.148:8000: connect: connection refused"
Oct 31 00:59:59.768301 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:59.768267    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"
Oct 31 00:59:59.869305 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:59.869258    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-home\" (UniqueName: \"kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-home\") pod \"1b410769-3505-4588-9255-fcca23401c3e\" (UID: \"1b410769-3505-4588-9255-fcca23401c3e\") "
Oct 31 00:59:59.869513 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:59.869370    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-model-cache\") pod \"1b410769-3505-4588-9255-fcca23401c3e\" (UID: \"1b410769-3505-4588-9255-fcca23401c3e\") "
Oct 31 00:59:59.869513 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:59.869403    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b410769-3505-4588-9255-fcca23401c3e-tls-certs\" (UniqueName: \"kubernetes.io/secret/1b410769-3505-4588-9255-fcca23401c3e-tls-certs\") pod \"1b410769-3505-4588-9255-fcca23401c3e\" (UID: \"1b410769-3505-4588-9255-fcca23401c3e\") "
Oct 31 00:59:59.869704 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:59.869604    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-tmp-dir\") pod \"1b410769-3505-4588-9255-fcca23401c3e\" (UID: \"1b410769-3505-4588-9255-fcca23401c3e\") "
Oct 31 00:59:59.869788 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:59.869753    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-model-cache" pod "1b410769-3505-4588-9255-fcca23401c3e" (UID: "1b410769-3505-4588-9255-fcca23401c3e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:59:59.869846 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:59.869761    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-dshm\") pod \"1b410769-3505-4588-9255-fcca23401c3e\" (UID: \"1b410769-3505-4588-9255-fcca23401c3e\") "
Oct 31 00:59:59.869966 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:59.869947    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b410769-3505-4588-9255-fcca23401c3e-kube-api-access-nsctq\" (UniqueName: \"kubernetes.io/projected/1b410769-3505-4588-9255-fcca23401c3e-kube-api-access-nsctq\") pod \"1b410769-3505-4588-9255-fcca23401c3e\" (UID: \"1b410769-3505-4588-9255-fcca23401c3e\") "
Oct 31 00:59:59.870044 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:59.869972    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-tmp-dir" pod "1b410769-3505-4588-9255-fcca23401c3e" (UID: "1b410769-3505-4588-9255-fcca23401c3e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:59:59.870784 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:59.870754    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-home" pod "1b410769-3505-4588-9255-fcca23401c3e" (UID: "1b410769-3505-4588-9255-fcca23401c3e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:59:59.870972 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:59.870865    4327 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-model-cache\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:59:59.870972 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:59.870888    4327 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-tmp-dir\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:59:59.873125 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:59.873095    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-dshm" pod "1b410769-3505-4588-9255-fcca23401c3e" (UID: "1b410769-3505-4588-9255-fcca23401c3e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:59:59.873240 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:59.873211    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b410769-3505-4588-9255-fcca23401c3e-kube-api-access-nsctq" pod "1b410769-3505-4588-9255-fcca23401c3e" (UID: "1b410769-3505-4588-9255-fcca23401c3e"). InnerVolumeSpecName "kube-api-access-nsctq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:59:59.873340 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:59.873324    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b410769-3505-4588-9255-fcca23401c3e-tls-certs" pod "1b410769-3505-4588-9255-fcca23401c3e" (UID: "1b410769-3505-4588-9255-fcca23401c3e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:59:59.972168 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:59.972136    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsctq\" (UniqueName: \"kubernetes.io/projected/1b410769-3505-4588-9255-fcca23401c3e-kube-api-access-nsctq\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:59:59.972168 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:59.972167    4327 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-home\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:59:59.972373 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:59.972177    4327 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b410769-3505-4588-9255-fcca23401c3e-tls-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 00:59:59.972373 ip-10-0-128-253 kubenswrapper[4327]: I1031 00:59:59.972186    4327 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b410769-3505-4588-9255-fcca23401c3e-dshm\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:00.212640 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.212610    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn"]
Oct 31 01:00:00.214039 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.214010    4327 cpu_manager.go:414] "containerMap: removing container" podUID="1b410769-3505-4588-9255-fcca23401c3e" containerName="main"
Oct 31 01:00:00.214039 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.214035    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b410769-3505-4588-9255-fcca23401c3e" containerName="main"
Oct 31 01:00:00.214323 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.214309    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b410769-3505-4588-9255-fcca23401c3e" containerName="main"
Oct 31 01:00:00.218325 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.218298    4327 generic.go:356] "Generic (PLEG): container finished" podID="1b410769-3505-4588-9255-fcca23401c3e" containerID="4e38930e5031186c751a3ddcba7d08966fd73b83238618c70e2d6acd0e856404" exitCode=0
Oct 31 01:00:00.218456 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.218401    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"
Oct 31 01:00:00.219232 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.219207    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6" event={"ID":"1b410769-3505-4588-9255-fcca23401c3e","Type":"ContainerDied","Data":"4e38930e5031186c751a3ddcba7d08966fd73b83238618c70e2d6acd0e856404"}
Oct 31 01:00:00.219337 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.219239    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6" event={"ID":"1b410769-3505-4588-9255-fcca23401c3e","Type":"ContainerDied","Data":"2b7072b23be82fbed85aaf297c4906fcf88d62759365a49e5c0cc931e094cb46"}
Oct 31 01:00:00.219337 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.219265    4327 scope.go:122] "RemoveContainer" containerID="4e38930e5031186c751a3ddcba7d08966fd73b83238618c70e2d6acd0e856404"
Oct 31 01:00:00.219337 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.219311    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn"
Oct 31 01:00:00.238482 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.238462    4327 scope.go:122] "RemoveContainer" containerID="4e38930e5031186c751a3ddcba7d08966fd73b83238618c70e2d6acd0e856404"
Oct 31 01:00:00.238838 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:00.238816    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e38930e5031186c751a3ddcba7d08966fd73b83238618c70e2d6acd0e856404\": container with ID starting with 4e38930e5031186c751a3ddcba7d08966fd73b83238618c70e2d6acd0e856404 not found: ID does not exist" containerID="4e38930e5031186c751a3ddcba7d08966fd73b83238618c70e2d6acd0e856404"
Oct 31 01:00:00.238922 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.238847    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e38930e5031186c751a3ddcba7d08966fd73b83238618c70e2d6acd0e856404"} err="failed to get container status \"4e38930e5031186c751a3ddcba7d08966fd73b83238618c70e2d6acd0e856404\": rpc error: code = NotFound desc = could not find container \"4e38930e5031186c751a3ddcba7d08966fd73b83238618c70e2d6acd0e856404\": container with ID starting with 4e38930e5031186c751a3ddcba7d08966fd73b83238618c70e2d6acd0e856404 not found: ID does not exist"
Oct 31 01:00:00.252810 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.252776    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn"]
Oct 31 01:00:00.262259 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.262224    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq"]
Oct 31 01:00:00.271844 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.271817    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq"
Oct 31 01:00:00.308907 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.308875    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\""
Oct 31 01:00:00.313763 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.313736    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t"]
Oct 31 01:00:00.318967 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.318944    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t"
Oct 31 01:00:00.320210 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.320186    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq"]
Oct 31 01:00:00.337824 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.337796    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7"]
Oct 31 01:00:00.350958 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.348406    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t"]
Oct 31 01:00:00.350958 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.348527    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7"
Oct 31 01:00:00.351516 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.351481    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7"]
Oct 31 01:00:00.357059 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.357037    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7"
Oct 31 01:00:00.378542 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.378514    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"]
Oct 31 01:00:00.379259 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.379230    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnljm\" (UniqueName: \"kubernetes.io/projected/c4ee6cb0-2737-426b-9f00-4d9f550c8164-kube-api-access-cnljm\") pod \"maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn\" (UID: \"c4ee6cb0-2737-426b-9f00-4d9f550c8164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn"
Oct 31 01:00:00.379460 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.379433    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ee6cb0-2737-426b-9f00-4d9f550c8164-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn\" (UID: \"c4ee6cb0-2737-426b-9f00-4d9f550c8164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn"
Oct 31 01:00:00.379612 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.379590    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11188116-a59b-48f1-9f7d-85d09032455e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq\" (UID: \"11188116-a59b-48f1-9f7d-85d09032455e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq"
Oct 31 01:00:00.379717 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.379636    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvzjh\" (UniqueName: \"kubernetes.io/projected/11188116-a59b-48f1-9f7d-85d09032455e-kube-api-access-cvzjh\") pod \"maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq\" (UID: \"11188116-a59b-48f1-9f7d-85d09032455e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq"
Oct 31 01:00:00.424083 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.416258    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm"]
Oct 31 01:00:00.431149 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.431121    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm"
Oct 31 01:00:00.443852 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.443808    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7"]
Oct 31 01:00:00.469363 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.469284    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7"]
Oct 31 01:00:00.491013 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.490982    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4d5621/test-model-4d5621-kserve-75549b8999-q24t6"]
Oct 31 01:00:00.491284 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.489020    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnljm\" (UniqueName: \"kubernetes.io/projected/c4ee6cb0-2737-426b-9f00-4d9f550c8164-kube-api-access-cnljm\") pod \"maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn\" (UID: \"c4ee6cb0-2737-426b-9f00-4d9f550c8164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn"
Oct 31 01:00:00.491451 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.491434    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmkng\" (UniqueName: \"kubernetes.io/projected/7f085592-c63f-433e-a77f-0b0c20746fe2-kube-api-access-tmkng\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t\" (UID: \"7f085592-c63f-433e-a77f-0b0c20746fe2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t"
Oct 31 01:00:00.491590 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.491567    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4htbn\" (UniqueName: \"kubernetes.io/projected/b118e1e7-3dd9-4107-b740-9032c60dee40-kube-api-access-4htbn\") pod \"maas-api-key-cleanup-29890140-k72t7\" (UID: \"b118e1e7-3dd9-4107-b740-9032c60dee40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7"
Oct 31 01:00:00.491853 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.491832    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b118e1e7-3dd9-4107-b740-9032c60dee40-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890140-k72t7\" (UID: \"b118e1e7-3dd9-4107-b740-9032c60dee40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7"
Oct 31 01:00:00.492011 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.491991    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13752887-b28f-49f8-855d-2eed4657dc61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7\" (UID: \"13752887-b28f-49f8-855d-2eed4657dc61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7"
Oct 31 01:00:00.492195 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.492179    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ee6cb0-2737-426b-9f00-4d9f550c8164-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn\" (UID: \"c4ee6cb0-2737-426b-9f00-4d9f550c8164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn"
Oct 31 01:00:00.492445 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.492427    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zwsk\" (UniqueName: \"kubernetes.io/projected/13752887-b28f-49f8-855d-2eed4657dc61-kube-api-access-6zwsk\") pod \"maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7\" (UID: \"13752887-b28f-49f8-855d-2eed4657dc61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7"
Oct 31 01:00:00.508196 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.508157    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11188116-a59b-48f1-9f7d-85d09032455e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq\" (UID: \"11188116-a59b-48f1-9f7d-85d09032455e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq"
Oct 31 01:00:00.518710 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.516097    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvzjh\" (UniqueName: \"kubernetes.io/projected/11188116-a59b-48f1-9f7d-85d09032455e-kube-api-access-cvzjh\") pod \"maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq\" (UID: \"11188116-a59b-48f1-9f7d-85d09032455e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq"
Oct 31 01:00:00.518710 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.516208    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f085592-c63f-433e-a77f-0b0c20746fe2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t\" (UID: \"7f085592-c63f-433e-a77f-0b0c20746fe2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t"
Oct 31 01:00:00.518710 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.495847    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ee6cb0-2737-426b-9f00-4d9f550c8164-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn\" (UID: \"c4ee6cb0-2737-426b-9f00-4d9f550c8164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn"
Oct 31 01:00:00.519206 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.519134    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11188116-a59b-48f1-9f7d-85d09032455e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq\" (UID: \"11188116-a59b-48f1-9f7d-85d09032455e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq"
Oct 31 01:00:00.554888 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.554852    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm"]
Oct 31 01:00:00.590235 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.590209    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt"]
Oct 31 01:00:00.591658 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.584569    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnljm\" (UniqueName: \"kubernetes.io/projected/c4ee6cb0-2737-426b-9f00-4d9f550c8164-kube-api-access-cnljm\") pod \"maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn\" (UID: \"c4ee6cb0-2737-426b-9f00-4d9f550c8164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn"
Oct 31 01:00:00.591658 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.581796    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvzjh\" (UniqueName: \"kubernetes.io/projected/11188116-a59b-48f1-9f7d-85d09032455e-kube-api-access-cvzjh\") pod \"maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq\" (UID: \"11188116-a59b-48f1-9f7d-85d09032455e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq"
Oct 31 01:00:00.597858 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.597762    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8"]
Oct 31 01:00:00.598020 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.597887    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt"
Oct 31 01:00:00.604221 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.604195    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8"
Oct 31 01:00:00.607775 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.607748    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt"]
Oct 31 01:00:00.615004 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.614983    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8"]
Oct 31 01:00:00.618495 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.617459    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpr94\" (UniqueName: \"kubernetes.io/projected/8b6d9a4f-7b06-42f1-8e32-720489b7ba93-kube-api-access-mpr94\") pod \"maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm\" (UID: \"8b6d9a4f-7b06-42f1-8e32-720489b7ba93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm"
Oct 31 01:00:00.618495 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.617534    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rndx\" (UniqueName: \"kubernetes.io/projected/f45f589e-69d1-49fd-b27d-a15d1f886235-kube-api-access-8rndx\") pod \"maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt\" (UID: \"f45f589e-69d1-49fd-b27d-a15d1f886235\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt"
Oct 31 01:00:00.618495 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.617573    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f085592-c63f-433e-a77f-0b0c20746fe2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t\" (UID: \"7f085592-c63f-433e-a77f-0b0c20746fe2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t"
Oct 31 01:00:00.618495 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.617594    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b6d9a4f-7b06-42f1-8e32-720489b7ba93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm\" (UID: \"8b6d9a4f-7b06-42f1-8e32-720489b7ba93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm"
Oct 31 01:00:00.618495 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.617660    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmkng\" (UniqueName: \"kubernetes.io/projected/7f085592-c63f-433e-a77f-0b0c20746fe2-kube-api-access-tmkng\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t\" (UID: \"7f085592-c63f-433e-a77f-0b0c20746fe2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t"
Oct 31 01:00:00.618495 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.617694    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4htbn\" (UniqueName: \"kubernetes.io/projected/b118e1e7-3dd9-4107-b740-9032c60dee40-kube-api-access-4htbn\") pod \"maas-api-key-cleanup-29890140-k72t7\" (UID: \"b118e1e7-3dd9-4107-b740-9032c60dee40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7"
Oct 31 01:00:00.618495 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.617760    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b118e1e7-3dd9-4107-b740-9032c60dee40-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890140-k72t7\" (UID: \"b118e1e7-3dd9-4107-b740-9032c60dee40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7"
Oct 31 01:00:00.618495 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.617781    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13752887-b28f-49f8-855d-2eed4657dc61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7\" (UID: \"13752887-b28f-49f8-855d-2eed4657dc61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7"
Oct 31 01:00:00.618495 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.617881    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45f589e-69d1-49fd-b27d-a15d1f886235-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt\" (UID: \"f45f589e-69d1-49fd-b27d-a15d1f886235\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt"
Oct 31 01:00:00.618495 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.617924    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zwsk\" (UniqueName: \"kubernetes.io/projected/13752887-b28f-49f8-855d-2eed4657dc61-kube-api-access-6zwsk\") pod \"maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7\" (UID: \"13752887-b28f-49f8-855d-2eed4657dc61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7"
Oct 31 01:00:00.619795 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.618889    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f085592-c63f-433e-a77f-0b0c20746fe2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t\" (UID: \"7f085592-c63f-433e-a77f-0b0c20746fe2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t"
Oct 31 01:00:00.619795 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.619631    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b118e1e7-3dd9-4107-b740-9032c60dee40-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890140-k72t7\" (UID: \"b118e1e7-3dd9-4107-b740-9032c60dee40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7"
Oct 31 01:00:00.620734 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.620114    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13752887-b28f-49f8-855d-2eed4657dc61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7\" (UID: \"13752887-b28f-49f8-855d-2eed4657dc61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7"
Oct 31 01:00:00.648092 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.648057    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zwsk\" (UniqueName: \"kubernetes.io/projected/13752887-b28f-49f8-855d-2eed4657dc61-kube-api-access-6zwsk\") pod \"maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7\" (UID: \"13752887-b28f-49f8-855d-2eed4657dc61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7"
Oct 31 01:00:00.648326 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.648301    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4htbn\" (UniqueName: \"kubernetes.io/projected/b118e1e7-3dd9-4107-b740-9032c60dee40-kube-api-access-4htbn\") pod \"maas-api-key-cleanup-29890140-k72t7\" (UID: \"b118e1e7-3dd9-4107-b740-9032c60dee40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7"
Oct 31 01:00:00.649304 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.649281    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmkng\" (UniqueName: \"kubernetes.io/projected/7f085592-c63f-433e-a77f-0b0c20746fe2-kube-api-access-tmkng\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t\" (UID: \"7f085592-c63f-433e-a77f-0b0c20746fe2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t"
Oct 31 01:00:00.655106 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.655081    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t"
Oct 31 01:00:00.673431 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.673407    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7"
Oct 31 01:00:00.692970 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.692929    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7"
Oct 31 01:00:00.721141 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.719609    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d352532d-5b9b-4713-a13c-32cd6d2c7716-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8\" (UID: \"d352532d-5b9b-4713-a13c-32cd6d2c7716\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8"
Oct 31 01:00:00.721141 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.719654    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpr94\" (UniqueName: \"kubernetes.io/projected/8b6d9a4f-7b06-42f1-8e32-720489b7ba93-kube-api-access-mpr94\") pod \"maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm\" (UID: \"8b6d9a4f-7b06-42f1-8e32-720489b7ba93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm"
Oct 31 01:00:00.721141 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.719753    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rndx\" (UniqueName: \"kubernetes.io/projected/f45f589e-69d1-49fd-b27d-a15d1f886235-kube-api-access-8rndx\") pod \"maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt\" (UID: \"f45f589e-69d1-49fd-b27d-a15d1f886235\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt"
Oct 31 01:00:00.721141 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.719802    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b6d9a4f-7b06-42f1-8e32-720489b7ba93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm\" (UID: \"8b6d9a4f-7b06-42f1-8e32-720489b7ba93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm"
Oct 31 01:00:00.721141 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.719844    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss9ph\" (UniqueName: \"kubernetes.io/projected/d352532d-5b9b-4713-a13c-32cd6d2c7716-kube-api-access-ss9ph\") pod \"maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8\" (UID: \"d352532d-5b9b-4713-a13c-32cd6d2c7716\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8"
Oct 31 01:00:00.721141 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.720022    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45f589e-69d1-49fd-b27d-a15d1f886235-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt\" (UID: \"f45f589e-69d1-49fd-b27d-a15d1f886235\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt"
Oct 31 01:00:00.721141 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.720580    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45f589e-69d1-49fd-b27d-a15d1f886235-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt\" (UID: \"f45f589e-69d1-49fd-b27d-a15d1f886235\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt"
Oct 31 01:00:00.721550 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.721388    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b6d9a4f-7b06-42f1-8e32-720489b7ba93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm\" (UID: \"8b6d9a4f-7b06-42f1-8e32-720489b7ba93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm"
Oct 31 01:00:00.753697 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.749765    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpr94\" (UniqueName: \"kubernetes.io/projected/8b6d9a4f-7b06-42f1-8e32-720489b7ba93-kube-api-access-mpr94\") pod \"maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm\" (UID: \"8b6d9a4f-7b06-42f1-8e32-720489b7ba93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm"
Oct 31 01:00:00.764227 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.764195    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rndx\" (UniqueName: \"kubernetes.io/projected/f45f589e-69d1-49fd-b27d-a15d1f886235-kube-api-access-8rndx\") pod \"maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt\" (UID: \"f45f589e-69d1-49fd-b27d-a15d1f886235\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt"
Oct 31 01:00:00.764516 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.764496    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm"
Oct 31 01:00:00.821459 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.821411    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d352532d-5b9b-4713-a13c-32cd6d2c7716-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8\" (UID: \"d352532d-5b9b-4713-a13c-32cd6d2c7716\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8"
Oct 31 01:00:00.821631 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.821614    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss9ph\" (UniqueName: \"kubernetes.io/projected/d352532d-5b9b-4713-a13c-32cd6d2c7716-kube-api-access-ss9ph\") pod \"maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8\" (UID: \"d352532d-5b9b-4713-a13c-32cd6d2c7716\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8"
Oct 31 01:00:00.822897 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.822872    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d352532d-5b9b-4713-a13c-32cd6d2c7716-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8\" (UID: \"d352532d-5b9b-4713-a13c-32cd6d2c7716\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8"
Oct 31 01:00:00.844637 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.843812    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn"
Oct 31 01:00:00.851579 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.851548    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss9ph\" (UniqueName: \"kubernetes.io/projected/d352532d-5b9b-4713-a13c-32cd6d2c7716-kube-api-access-ss9ph\") pod \"maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8\" (UID: \"d352532d-5b9b-4713-a13c-32cd6d2c7716\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8"
Oct 31 01:00:00.889647 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.889384    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq"
Oct 31 01:00:00.931482 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.931456    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt"
Oct 31 01:00:00.938522 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.938495    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8"
Oct 31 01:00:00.987060 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:00.987024    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t"]
Oct 31 01:00:01.022609 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:01.019982    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7"]
Oct 31 01:00:01.080628 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:01.070723    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7"]
Oct 31 01:00:01.080628 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:00:01.079911    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb118e1e7_3dd9_4107_b740_9032c60dee40.slice/crio-209d7bc2ac64cc962c365a0e9816338e4164e8dc66c03af8041a492614f87563 WatchSource:0}: Error finding container 209d7bc2ac64cc962c365a0e9816338e4164e8dc66c03af8041a492614f87563: Status 404 returned error can't find the container with id 209d7bc2ac64cc962c365a0e9816338e4164e8dc66c03af8041a492614f87563
Oct 31 01:00:01.133309 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:01.133286    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm"]
Oct 31 01:00:01.133450 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:00:01.133372    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b6d9a4f_7b06_42f1_8e32_720489b7ba93.slice/crio-5e5805e2d055163a00f5a03cd97818fa3acf3ea3f7ca6d4b73ab6505789010b2 WatchSource:0}: Error finding container 5e5805e2d055163a00f5a03cd97818fa3acf3ea3f7ca6d4b73ab6505789010b2: Status 404 returned error can't find the container with id 5e5805e2d055163a00f5a03cd97818fa3acf3ea3f7ca6d4b73ab6505789010b2
Oct 31 01:00:01.198563 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:00:01.198530    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ee6cb0_2737_426b_9f00_4d9f550c8164.slice/crio-fc500358463fd890b8b3bf071778492c46b0941b61b75bda8986664b9f74d23e WatchSource:0}: Error finding container fc500358463fd890b8b3bf071778492c46b0941b61b75bda8986664b9f74d23e: Status 404 returned error can't find the container with id fc500358463fd890b8b3bf071778492c46b0941b61b75bda8986664b9f74d23e
Oct 31 01:00:01.198721 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:01.198617    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn"]
Oct 31 01:00:01.230237 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:01.230161    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7" event={"ID":"b118e1e7-3dd9-4107-b740-9032c60dee40","Type":"ContainerStarted","Data":"209d7bc2ac64cc962c365a0e9816338e4164e8dc66c03af8041a492614f87563"}
Oct 31 01:00:01.231625 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:01.231588    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn" event={"ID":"c4ee6cb0-2737-426b-9f00-4d9f550c8164","Type":"ContainerStarted","Data":"fc500358463fd890b8b3bf071778492c46b0941b61b75bda8986664b9f74d23e"}
Oct 31 01:00:01.232723 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:01.232696    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t" event={"ID":"7f085592-c63f-433e-a77f-0b0c20746fe2","Type":"ContainerStarted","Data":"5c4c65b5492a1bf35b064c91519f917165d95548c109c58457655282d0554cc0"}
Oct 31 01:00:01.234560 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:01.234537    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm" event={"ID":"8b6d9a4f-7b06-42f1-8e32-720489b7ba93","Type":"ContainerStarted","Data":"5e5805e2d055163a00f5a03cd97818fa3acf3ea3f7ca6d4b73ab6505789010b2"}
Oct 31 01:00:01.235722 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:01.235699    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7" event={"ID":"13752887-b28f-49f8-855d-2eed4657dc61","Type":"ContainerStarted","Data":"7f6f75a511fb25fd16231bf9f5253f3fedc6eb5c8b99440538e348cd980188e1"}
Oct 31 01:00:01.344897 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:00:01.344871    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd352532d_5b9b_4713_a13c_32cd6d2c7716.slice/crio-6790918b4979ef31ebaf3f51243e5438403ed845fb72f65f233f53e1ba65ec89 WatchSource:0}: Error finding container 6790918b4979ef31ebaf3f51243e5438403ed845fb72f65f233f53e1ba65ec89: Status 404 returned error can't find the container with id 6790918b4979ef31ebaf3f51243e5438403ed845fb72f65f233f53e1ba65ec89
Oct 31 01:00:01.346151 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:01.346129    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8"]
Oct 31 01:00:01.446935 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:01.446906    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq"]
Oct 31 01:00:01.447060 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:00:01.446963    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11188116_a59b_48f1_9f7d_85d09032455e.slice/crio-1f6740b6efb80f8d219a129a64ef90600ffeb092548b3a251af4dc268f06657d WatchSource:0}: Error finding container 1f6740b6efb80f8d219a129a64ef90600ffeb092548b3a251af4dc268f06657d: Status 404 returned error can't find the container with id 1f6740b6efb80f8d219a129a64ef90600ffeb092548b3a251af4dc268f06657d
Oct 31 01:00:01.492001 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:00:01.491976    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45f589e_69d1_49fd_b27d_a15d1f886235.slice/crio-8d8f099eac1c117bb2f335b7568f81d9fe50b851ea417422eea2847fa96e72a0 WatchSource:0}: Error finding container 8d8f099eac1c117bb2f335b7568f81d9fe50b851ea417422eea2847fa96e72a0: Status 404 returned error can't find the container with id 8d8f099eac1c117bb2f335b7568f81d9fe50b851ea417422eea2847fa96e72a0
Oct 31 01:00:01.492549 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:01.492026    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt"]
Oct 31 01:00:01.929031 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:01.929001    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b410769-3505-4588-9255-fcca23401c3e" path="/var/lib/kubelet/pods/1b410769-3505-4588-9255-fcca23401c3e/volumes"
Oct 31 01:00:02.246813 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:02.246774    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7" event={"ID":"b118e1e7-3dd9-4107-b740-9032c60dee40","Type":"ContainerStarted","Data":"c656119b2cf2f5d40b1b8324fe3104359c9233087dbd459881324241f2aaec4d"}
Oct 31 01:00:02.248850 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:02.248820    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn" event={"ID":"c4ee6cb0-2737-426b-9f00-4d9f550c8164","Type":"ContainerStarted","Data":"7d97c215adfbb6cdc6f4a023f54be59d999197e45446040770ee9be30a7635bd"}
Oct 31 01:00:02.252251 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:02.252225    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t" event={"ID":"7f085592-c63f-433e-a77f-0b0c20746fe2","Type":"ContainerStarted","Data":"d42a41f796be0dce2924e5ffceb7cdabb93dc1f7a6b391c0ce4a1b8f35e1073a"}
Oct 31 01:00:02.253874 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:02.253848    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt" event={"ID":"f45f589e-69d1-49fd-b27d-a15d1f886235","Type":"ContainerStarted","Data":"9c64b6aba5621c2b715d1bf92ac739ee6e138f80c7759a0f36f698a2c995c82c"}
Oct 31 01:00:02.253994 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:02.253898    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt" event={"ID":"f45f589e-69d1-49fd-b27d-a15d1f886235","Type":"ContainerStarted","Data":"8d8f099eac1c117bb2f335b7568f81d9fe50b851ea417422eea2847fa96e72a0"}
Oct 31 01:00:02.255513 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:02.255489    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq" event={"ID":"11188116-a59b-48f1-9f7d-85d09032455e","Type":"ContainerStarted","Data":"db27670031d66cc52c3caa157ab7782538b87b7a9fece7b65f6973ed001e045a"}
Oct 31 01:00:02.255599 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:02.255539    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq" event={"ID":"11188116-a59b-48f1-9f7d-85d09032455e","Type":"ContainerStarted","Data":"1f6740b6efb80f8d219a129a64ef90600ffeb092548b3a251af4dc268f06657d"}
Oct 31 01:00:02.258002 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:02.257968    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm" event={"ID":"8b6d9a4f-7b06-42f1-8e32-720489b7ba93","Type":"ContainerStarted","Data":"e70925b0239e82924a9e6dac137e3551913e7025583f8bce58cfcdf3a1ab5187"}
Oct 31 01:00:02.259646 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:02.259623    4327 generic.go:356] "Generic (PLEG): container finished" podID="13752887-b28f-49f8-855d-2eed4657dc61" containerID="c5d111c31173efbbbefebe8ad1d21dca569c77fa504e9e95694a75fde3798dcc" exitCode=60
Oct 31 01:00:02.259749 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:02.259707    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7" event={"ID":"13752887-b28f-49f8-855d-2eed4657dc61","Type":"ContainerDied","Data":"c5d111c31173efbbbefebe8ad1d21dca569c77fa504e9e95694a75fde3798dcc"}
Oct 31 01:00:02.259944 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:02.259929    4327 scope.go:122] "RemoveContainer" containerID="c5d111c31173efbbbefebe8ad1d21dca569c77fa504e9e95694a75fde3798dcc"
Oct 31 01:00:02.261631 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:02.261503    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8" event={"ID":"d352532d-5b9b-4713-a13c-32cd6d2c7716","Type":"ContainerStarted","Data":"7fda0449926bfbbb4c07a26d42631d8f112824b61009ffed4f9ecb5e3da9b58a"}
Oct 31 01:00:02.261631 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:02.261547    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8" event={"ID":"d352532d-5b9b-4713-a13c-32cd6d2c7716","Type":"ContainerStarted","Data":"6790918b4979ef31ebaf3f51243e5438403ed845fb72f65f233f53e1ba65ec89"}
Oct 31 01:00:02.291848 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:02.291797    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7" podStartSLOduration=2.291780095 podStartE2EDuration="2.291780095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.287083581 +0000 UTC m=+2462.509119123" watchObservedRunningTime="2026-10-31 01:00:02.291780095 +0000 UTC m=+2462.513815637"
Oct 31 01:00:02.381399 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:02.381338    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn" podStartSLOduration=2.381323336 podStartE2EDuration="2.381323336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.336324324 +0000 UTC m=+2462.558359866" watchObservedRunningTime="2026-10-31 01:00:02.381323336 +0000 UTC m=+2462.603358926"
Oct 31 01:00:02.500801 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:02.500735    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt" podStartSLOduration=2.500714313 podStartE2EDuration="2.500714313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.467746048 +0000 UTC m=+2462.689781585" watchObservedRunningTime="2026-10-31 01:00:02.500714313 +0000 UTC m=+2462.722749855"
Oct 31 01:00:02.532058 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:02.531957    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq" podStartSLOduration=2.531936828 podStartE2EDuration="2.531936828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.529046613 +0000 UTC m=+2462.751082155" watchObservedRunningTime="2026-10-31 01:00:02.531936828 +0000 UTC m=+2462.753972369"
Oct 31 01:00:02.572435 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:02.572383    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm" podStartSLOduration=2.572367861 podStartE2EDuration="2.572367861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.569476975 +0000 UTC m=+2462.791512531" watchObservedRunningTime="2026-10-31 01:00:02.572367861 +0000 UTC m=+2462.794403382"
Oct 31 01:00:02.615810 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:02.615760    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8" podStartSLOduration=2.6157464040000002 podStartE2EDuration="2.615746404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.613157031 +0000 UTC m=+2462.835192572" watchObservedRunningTime="2026-10-31 01:00:02.615746404 +0000 UTC m=+2462.837781946"
Oct 31 01:00:02.657482 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:02.657428    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t" podStartSLOduration=2.657412361 podStartE2EDuration="2.657412361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.653398082 +0000 UTC m=+2462.875433640" watchObservedRunningTime="2026-10-31 01:00:02.657412361 +0000 UTC m=+2462.879447946"
Oct 31 01:00:03.273052 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:03.273014    4327 generic.go:356] "Generic (PLEG): container finished" podID="b118e1e7-3dd9-4107-b740-9032c60dee40" containerID="c656119b2cf2f5d40b1b8324fe3104359c9233087dbd459881324241f2aaec4d" exitCode=60
Oct 31 01:00:03.273243 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:03.273097    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7" event={"ID":"b118e1e7-3dd9-4107-b740-9032c60dee40","Type":"ContainerDied","Data":"c656119b2cf2f5d40b1b8324fe3104359c9233087dbd459881324241f2aaec4d"}
Oct 31 01:00:03.273534 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:03.273493    4327 scope.go:122] "RemoveContainer" containerID="c656119b2cf2f5d40b1b8324fe3104359c9233087dbd459881324241f2aaec4d"
Oct 31 01:00:03.275002 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:03.274976    4327 generic.go:356] "Generic (PLEG): container finished" podID="c4ee6cb0-2737-426b-9f00-4d9f550c8164" containerID="7d97c215adfbb6cdc6f4a023f54be59d999197e45446040770ee9be30a7635bd" exitCode=60
Oct 31 01:00:03.275125 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:03.275059    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn" event={"ID":"c4ee6cb0-2737-426b-9f00-4d9f550c8164","Type":"ContainerDied","Data":"7d97c215adfbb6cdc6f4a023f54be59d999197e45446040770ee9be30a7635bd"}
Oct 31 01:00:03.275389 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:03.275367    4327 scope.go:122] "RemoveContainer" containerID="7d97c215adfbb6cdc6f4a023f54be59d999197e45446040770ee9be30a7635bd"
Oct 31 01:00:03.277563 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:03.277539    4327 generic.go:356] "Generic (PLEG): container finished" podID="7f085592-c63f-433e-a77f-0b0c20746fe2" containerID="d42a41f796be0dce2924e5ffceb7cdabb93dc1f7a6b391c0ce4a1b8f35e1073a" exitCode=60
Oct 31 01:00:03.277667 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:03.277567    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t" event={"ID":"7f085592-c63f-433e-a77f-0b0c20746fe2","Type":"ContainerDied","Data":"d42a41f796be0dce2924e5ffceb7cdabb93dc1f7a6b391c0ce4a1b8f35e1073a"}
Oct 31 01:00:03.278012 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:03.277871    4327 scope.go:122] "RemoveContainer" containerID="d42a41f796be0dce2924e5ffceb7cdabb93dc1f7a6b391c0ce4a1b8f35e1073a"
Oct 31 01:00:03.279896 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:03.279873    4327 generic.go:356] "Generic (PLEG): container finished" podID="f45f589e-69d1-49fd-b27d-a15d1f886235" containerID="9c64b6aba5621c2b715d1bf92ac739ee6e138f80c7759a0f36f698a2c995c82c" exitCode=60
Oct 31 01:00:03.280005 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:03.279962    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt" event={"ID":"f45f589e-69d1-49fd-b27d-a15d1f886235","Type":"ContainerDied","Data":"9c64b6aba5621c2b715d1bf92ac739ee6e138f80c7759a0f36f698a2c995c82c"}
Oct 31 01:00:03.280193 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:03.280172    4327 scope.go:122] "RemoveContainer" containerID="9c64b6aba5621c2b715d1bf92ac739ee6e138f80c7759a0f36f698a2c995c82c"
Oct 31 01:00:03.282027 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:03.282004    4327 generic.go:356] "Generic (PLEG): container finished" podID="11188116-a59b-48f1-9f7d-85d09032455e" containerID="db27670031d66cc52c3caa157ab7782538b87b7a9fece7b65f6973ed001e045a" exitCode=60
Oct 31 01:00:03.282142 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:03.282086    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq" event={"ID":"11188116-a59b-48f1-9f7d-85d09032455e","Type":"ContainerDied","Data":"db27670031d66cc52c3caa157ab7782538b87b7a9fece7b65f6973ed001e045a"}
Oct 31 01:00:03.282292 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:03.282277    4327 scope.go:122] "RemoveContainer" containerID="db27670031d66cc52c3caa157ab7782538b87b7a9fece7b65f6973ed001e045a"
Oct 31 01:00:03.283848 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:03.283826    4327 generic.go:356] "Generic (PLEG): container finished" podID="8b6d9a4f-7b06-42f1-8e32-720489b7ba93" containerID="e70925b0239e82924a9e6dac137e3551913e7025583f8bce58cfcdf3a1ab5187" exitCode=60
Oct 31 01:00:03.283968 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:03.283849    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm" event={"ID":"8b6d9a4f-7b06-42f1-8e32-720489b7ba93","Type":"ContainerDied","Data":"e70925b0239e82924a9e6dac137e3551913e7025583f8bce58cfcdf3a1ab5187"}
Oct 31 01:00:03.284135 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:03.284116    4327 scope.go:122] "RemoveContainer" containerID="e70925b0239e82924a9e6dac137e3551913e7025583f8bce58cfcdf3a1ab5187"
Oct 31 01:00:03.286001 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:03.285971    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7" event={"ID":"13752887-b28f-49f8-855d-2eed4657dc61","Type":"ContainerStarted","Data":"998b2ec9486e13c9b45903d76962c562c3ee8d9cc5702164ff8b76ee72306bfe"}
Oct 31 01:00:03.288448 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:03.288423    4327 generic.go:356] "Generic (PLEG): container finished" podID="d352532d-5b9b-4713-a13c-32cd6d2c7716" containerID="7fda0449926bfbbb4c07a26d42631d8f112824b61009ffed4f9ecb5e3da9b58a" exitCode=60
Oct 31 01:00:03.288519 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:03.288503    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8" event={"ID":"d352532d-5b9b-4713-a13c-32cd6d2c7716","Type":"ContainerDied","Data":"7fda0449926bfbbb4c07a26d42631d8f112824b61009ffed4f9ecb5e3da9b58a"}
Oct 31 01:00:03.288772 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:03.288754    4327 scope.go:122] "RemoveContainer" containerID="7fda0449926bfbbb4c07a26d42631d8f112824b61009ffed4f9ecb5e3da9b58a"
Oct 31 01:00:03.608902 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:03.608849    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7" podStartSLOduration=3.608835273 podStartE2EDuration="3.608835273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:03.606385918 +0000 UTC m=+2463.828421460" watchObservedRunningTime="2026-10-31 01:00:03.608835273 +0000 UTC m=+2463.830870814"
Oct 31 01:00:04.330108 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.330073    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn" event={"ID":"c4ee6cb0-2737-426b-9f00-4d9f550c8164","Type":"ContainerStarted","Data":"b812e9c53991ed4940d9f843b4344eacb0f7284314a126576c42b8bc9575606b"}
Oct 31 01:00:04.332780 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.332738    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t" event={"ID":"7f085592-c63f-433e-a77f-0b0c20746fe2","Type":"ContainerStarted","Data":"3af96efdce87e5eea99bd030c8317493fb03d07a9828ce2a53772849c0d92b42"}
Oct 31 01:00:04.335233 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.335201    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt" event={"ID":"f45f589e-69d1-49fd-b27d-a15d1f886235","Type":"ContainerStarted","Data":"61c9a8293ae387d46c835ebe3e6b03b0891a4bd225da234632b8ea0e62549a5d"}
Oct 31 01:00:04.335606 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.335582    4327 scope.go:122] "RemoveContainer" containerID="61c9a8293ae387d46c835ebe3e6b03b0891a4bd225da234632b8ea0e62549a5d"
Oct 31 01:00:04.336075 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:04.336049    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt_odh-ai-gateway-infra(f45f589e-69d1-49fd-b27d-a15d1f886235)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt" podUID="f45f589e-69d1-49fd-b27d-a15d1f886235"
Oct 31 01:00:04.337459 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.337435    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq" event={"ID":"11188116-a59b-48f1-9f7d-85d09032455e","Type":"ContainerStarted","Data":"5a3c3d924779efce8509d3d576c9456586a4adb2fd7d9afa91879f03b25eb75b"}
Oct 31 01:00:04.339579 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.339555    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm" event={"ID":"8b6d9a4f-7b06-42f1-8e32-720489b7ba93","Type":"ContainerStarted","Data":"0815f5e78b90ea10782475d318e3c6f4a5e7e4faf21c988cb3aa7474fca50e07"}
Oct 31 01:00:04.341503 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.341481    4327 generic.go:356] "Generic (PLEG): container finished" podID="13752887-b28f-49f8-855d-2eed4657dc61" containerID="998b2ec9486e13c9b45903d76962c562c3ee8d9cc5702164ff8b76ee72306bfe" exitCode=60
Oct 31 01:00:04.341602 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.341553    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7" event={"ID":"13752887-b28f-49f8-855d-2eed4657dc61","Type":"ContainerDied","Data":"998b2ec9486e13c9b45903d76962c562c3ee8d9cc5702164ff8b76ee72306bfe"}
Oct 31 01:00:04.341722 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.341609    4327 scope.go:122] "RemoveContainer" containerID="c5d111c31173efbbbefebe8ad1d21dca569c77fa504e9e95694a75fde3798dcc"
Oct 31 01:00:04.341827 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.341808    4327 scope.go:122] "RemoveContainer" containerID="998b2ec9486e13c9b45903d76962c562c3ee8d9cc5702164ff8b76ee72306bfe"
Oct 31 01:00:04.342177 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:04.342155    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7_odh-ai-gateway-infra(13752887-b28f-49f8-855d-2eed4657dc61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7" podUID="13752887-b28f-49f8-855d-2eed4657dc61"
Oct 31 01:00:04.343936 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.343910    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8" event={"ID":"d352532d-5b9b-4713-a13c-32cd6d2c7716","Type":"ContainerStarted","Data":"958f370c31a0097a783153a04c95c10245e1971ac94b868ef40ce480d8b64d4c"}
Oct 31 01:00:04.349653 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.349634    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7" event={"ID":"b118e1e7-3dd9-4107-b740-9032c60dee40","Type":"ContainerStarted","Data":"ebd0755e1ffe5f26b978139b6edd66e49ada20150ebd29b1a4cb104f905845a6"}
Oct 31 01:00:04.408620 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.408582    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"]
Oct 31 01:00:04.416821 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.416793    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.434929 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.434899    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-82b5e8-openshift-default-dockercfg-tlr4n\""
Oct 31 01:00:04.442244 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.442216    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"]
Oct 31 01:00:04.448008 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:04.444232    4327 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ee6cb0_2737_426b_9f00_4d9f550c8164.slice/crio-b812e9c53991ed4940d9f843b4344eacb0f7284314a126576c42b8bc9575606b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45f589e_69d1_49fd_b27d_a15d1f886235.slice/crio-conmon-61c9a8293ae387d46c835ebe3e6b03b0891a4bd225da234632b8ea0e62549a5d.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 01:00:04.507605 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.507564    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-workload-socket\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.507791 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.507743    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-credential-socket\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.507874 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.507795    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a206da9-cfc0-41d4-888c-1123aab28e5c-istiod-ca-cert\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.507874 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.507838    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b2qw\" (UniqueName: \"kubernetes.io/projected/7a206da9-cfc0-41d4-888c-1123aab28e5c-kube-api-access-9b2qw\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.508038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.508019    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-envoy\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.508096 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.508056    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-workload-certs\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.508193 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.508173    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-data\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.508260 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.508205    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-podinfo\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.508355 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.508331    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-token\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.610724 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.610653    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-workload-socket\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.611136 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.610784    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-credential-socket\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.611136 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.610857    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a206da9-cfc0-41d4-888c-1123aab28e5c-istiod-ca-cert\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.611136 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.610888    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b2qw\" (UniqueName: \"kubernetes.io/projected/7a206da9-cfc0-41d4-888c-1123aab28e5c-kube-api-access-9b2qw\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.611136 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.611034    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-envoy\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.611136 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.611084    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-workload-certs\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.611136 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.611117    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-workload-socket\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.611474 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.611128    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-credential-socket\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.611474 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.611178    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-data\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.611474 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.611204    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-podinfo\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.611474 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.611259    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-token\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.611474 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.611400    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-workload-certs\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.611764 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.611483    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-data\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.611764 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.611587    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a206da9-cfc0-41d4-888c-1123aab28e5c-istiod-ca-cert\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.613339 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.613312    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-envoy\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.613691 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.613658    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-podinfo\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.637953 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.637922    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-token\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.638051 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.637995    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b2qw\" (UniqueName: \"kubernetes.io/projected/7a206da9-cfc0-41d4-888c-1123aab28e5c-kube-api-access-9b2qw\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:04.805231 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:04.805203    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:05.136042 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:00:05.136009    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a206da9_cfc0_41d4_888c_1123aab28e5c.slice/crio-6fd36b749f73c18ed613cc186e341ebf61c6e12edf8ea34c2335cc053b79dce7 WatchSource:0}: Error finding container 6fd36b749f73c18ed613cc186e341ebf61c6e12edf8ea34c2335cc053b79dce7: Status 404 returned error can't find the container with id 6fd36b749f73c18ed613cc186e341ebf61c6e12edf8ea34c2335cc053b79dce7
Oct 31 01:00:05.136527 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.136507    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"]
Oct 31 01:00:05.143478 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.143409    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:00:05.143608 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.143525    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:00:05.143608 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.143569    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:00:05.360647 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.360611    4327 generic.go:356] "Generic (PLEG): container finished" podID="c4ee6cb0-2737-426b-9f00-4d9f550c8164" containerID="b812e9c53991ed4940d9f843b4344eacb0f7284314a126576c42b8bc9575606b" exitCode=60
Oct 31 01:00:05.360846 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.360721    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn" event={"ID":"c4ee6cb0-2737-426b-9f00-4d9f550c8164","Type":"ContainerDied","Data":"b812e9c53991ed4940d9f843b4344eacb0f7284314a126576c42b8bc9575606b"}
Oct 31 01:00:05.360846 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.360824    4327 scope.go:122] "RemoveContainer" containerID="7d97c215adfbb6cdc6f4a023f54be59d999197e45446040770ee9be30a7635bd"
Oct 31 01:00:05.361253 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.361231    4327 scope.go:122] "RemoveContainer" containerID="b812e9c53991ed4940d9f843b4344eacb0f7284314a126576c42b8bc9575606b"
Oct 31 01:00:05.361691 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:05.361651    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn_odh-ai-gateway-infra(c4ee6cb0-2737-426b-9f00-4d9f550c8164)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn" podUID="c4ee6cb0-2737-426b-9f00-4d9f550c8164"
Oct 31 01:00:05.363645 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.363623    4327 generic.go:356] "Generic (PLEG): container finished" podID="7f085592-c63f-433e-a77f-0b0c20746fe2" containerID="3af96efdce87e5eea99bd030c8317493fb03d07a9828ce2a53772849c0d92b42" exitCode=60
Oct 31 01:00:05.363748 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.363732    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t" event={"ID":"7f085592-c63f-433e-a77f-0b0c20746fe2","Type":"ContainerDied","Data":"3af96efdce87e5eea99bd030c8317493fb03d07a9828ce2a53772849c0d92b42"}
Oct 31 01:00:05.364144 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.364120    4327 scope.go:122] "RemoveContainer" containerID="3af96efdce87e5eea99bd030c8317493fb03d07a9828ce2a53772849c0d92b42"
Oct 31 01:00:05.364609 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:05.364585    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t_odh-ai-gateway-infra(7f085592-c63f-433e-a77f-0b0c20746fe2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t" podUID="7f085592-c63f-433e-a77f-0b0c20746fe2"
Oct 31 01:00:05.366088 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.366062    4327 generic.go:356] "Generic (PLEG): container finished" podID="f45f589e-69d1-49fd-b27d-a15d1f886235" containerID="61c9a8293ae387d46c835ebe3e6b03b0891a4bd225da234632b8ea0e62549a5d" exitCode=60
Oct 31 01:00:05.366194 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.366155    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt" event={"ID":"f45f589e-69d1-49fd-b27d-a15d1f886235","Type":"ContainerDied","Data":"61c9a8293ae387d46c835ebe3e6b03b0891a4bd225da234632b8ea0e62549a5d"}
Oct 31 01:00:05.366536 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.366499    4327 scope.go:122] "RemoveContainer" containerID="61c9a8293ae387d46c835ebe3e6b03b0891a4bd225da234632b8ea0e62549a5d"
Oct 31 01:00:05.366918 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:05.366889    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt_odh-ai-gateway-infra(f45f589e-69d1-49fd-b27d-a15d1f886235)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt" podUID="f45f589e-69d1-49fd-b27d-a15d1f886235"
Oct 31 01:00:05.368551 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.368520    4327 generic.go:356] "Generic (PLEG): container finished" podID="11188116-a59b-48f1-9f7d-85d09032455e" containerID="5a3c3d924779efce8509d3d576c9456586a4adb2fd7d9afa91879f03b25eb75b" exitCode=60
Oct 31 01:00:05.368643 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.368570    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq" event={"ID":"11188116-a59b-48f1-9f7d-85d09032455e","Type":"ContainerDied","Data":"5a3c3d924779efce8509d3d576c9456586a4adb2fd7d9afa91879f03b25eb75b"}
Oct 31 01:00:05.368990 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.368969    4327 scope.go:122] "RemoveContainer" containerID="5a3c3d924779efce8509d3d576c9456586a4adb2fd7d9afa91879f03b25eb75b"
Oct 31 01:00:05.369353 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:05.369327    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq_odh-ai-gateway-infra(11188116-a59b-48f1-9f7d-85d09032455e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq" podUID="11188116-a59b-48f1-9f7d-85d09032455e"
Oct 31 01:00:05.372267 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.372246    4327 generic.go:356] "Generic (PLEG): container finished" podID="8b6d9a4f-7b06-42f1-8e32-720489b7ba93" containerID="0815f5e78b90ea10782475d318e3c6f4a5e7e4faf21c988cb3aa7474fca50e07" exitCode=60
Oct 31 01:00:05.372355 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.372312    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm" event={"ID":"8b6d9a4f-7b06-42f1-8e32-720489b7ba93","Type":"ContainerDied","Data":"0815f5e78b90ea10782475d318e3c6f4a5e7e4faf21c988cb3aa7474fca50e07"}
Oct 31 01:00:05.372696 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.372667    4327 scope.go:122] "RemoveContainer" containerID="0815f5e78b90ea10782475d318e3c6f4a5e7e4faf21c988cb3aa7474fca50e07"
Oct 31 01:00:05.373117 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:05.373094    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm_odh-ai-gateway-infra(8b6d9a4f-7b06-42f1-8e32-720489b7ba93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm" podUID="8b6d9a4f-7b06-42f1-8e32-720489b7ba93"
Oct 31 01:00:05.374907 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.374887    4327 scope.go:122] "RemoveContainer" containerID="998b2ec9486e13c9b45903d76962c562c3ee8d9cc5702164ff8b76ee72306bfe"
Oct 31 01:00:05.375238 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:05.375212    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7_odh-ai-gateway-infra(13752887-b28f-49f8-855d-2eed4657dc61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7" podUID="13752887-b28f-49f8-855d-2eed4657dc61"
Oct 31 01:00:05.378798 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.378780    4327 scope.go:122] "RemoveContainer" containerID="d42a41f796be0dce2924e5ffceb7cdabb93dc1f7a6b391c0ce4a1b8f35e1073a"
Oct 31 01:00:05.379649 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.379624    4327 generic.go:356] "Generic (PLEG): container finished" podID="d352532d-5b9b-4713-a13c-32cd6d2c7716" containerID="958f370c31a0097a783153a04c95c10245e1971ac94b868ef40ce480d8b64d4c" exitCode=60
Oct 31 01:00:05.379752 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.379726    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8" event={"ID":"d352532d-5b9b-4713-a13c-32cd6d2c7716","Type":"ContainerDied","Data":"958f370c31a0097a783153a04c95c10245e1971ac94b868ef40ce480d8b64d4c"}
Oct 31 01:00:05.379994 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.379963    4327 scope.go:122] "RemoveContainer" containerID="958f370c31a0097a783153a04c95c10245e1971ac94b868ef40ce480d8b64d4c"
Oct 31 01:00:05.380237 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:05.380214    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8_odh-ai-gateway-infra(d352532d-5b9b-4713-a13c-32cd6d2c7716)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8" podUID="d352532d-5b9b-4713-a13c-32cd6d2c7716"
Oct 31 01:00:05.381753 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.381727    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65" event={"ID":"7a206da9-cfc0-41d4-888c-1123aab28e5c","Type":"ContainerStarted","Data":"4f90bb472df3a4a3cb5967b8be094dfa52ae382eddd6cc82b4b2f4d108f910ab"}
Oct 31 01:00:05.381843 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.381782    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65" event={"ID":"7a206da9-cfc0-41d4-888c-1123aab28e5c","Type":"ContainerStarted","Data":"6fd36b749f73c18ed613cc186e341ebf61c6e12edf8ea34c2335cc053b79dce7"}
Oct 31 01:00:05.383768 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.383669    4327 generic.go:356] "Generic (PLEG): container finished" podID="b118e1e7-3dd9-4107-b740-9032c60dee40" containerID="ebd0755e1ffe5f26b978139b6edd66e49ada20150ebd29b1a4cb104f905845a6" exitCode=60
Oct 31 01:00:05.383856 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.383761    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7" event={"ID":"b118e1e7-3dd9-4107-b740-9032c60dee40","Type":"ContainerDied","Data":"ebd0755e1ffe5f26b978139b6edd66e49ada20150ebd29b1a4cb104f905845a6"}
Oct 31 01:00:05.384015 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.384000    4327 scope.go:122] "RemoveContainer" containerID="ebd0755e1ffe5f26b978139b6edd66e49ada20150ebd29b1a4cb104f905845a6"
Oct 31 01:00:05.384245 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:05.384228    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890140-k72t7_odh-ai-gateway-infra(b118e1e7-3dd9-4107-b740-9032c60dee40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7" podUID="b118e1e7-3dd9-4107-b740-9032c60dee40"
Oct 31 01:00:05.396149 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.396130    4327 scope.go:122] "RemoveContainer" containerID="9c64b6aba5621c2b715d1bf92ac739ee6e138f80c7759a0f36f698a2c995c82c"
Oct 31 01:00:05.419168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.419089    4327 scope.go:122] "RemoveContainer" containerID="db27670031d66cc52c3caa157ab7782538b87b7a9fece7b65f6973ed001e045a"
Oct 31 01:00:05.438449 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.438422    4327 scope.go:122] "RemoveContainer" containerID="e70925b0239e82924a9e6dac137e3551913e7025583f8bce58cfcdf3a1ab5187"
Oct 31 01:00:05.454117 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.454090    4327 scope.go:122] "RemoveContainer" containerID="7fda0449926bfbbb4c07a26d42631d8f112824b61009ffed4f9ecb5e3da9b58a"
Oct 31 01:00:05.470898 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.470875    4327 scope.go:122] "RemoveContainer" containerID="c656119b2cf2f5d40b1b8324fe3104359c9233087dbd459881324241f2aaec4d"
Oct 31 01:00:05.762038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.761931    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65" podStartSLOduration=1.761916496 podStartE2EDuration="1.761916496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:05.75984963 +0000 UTC m=+2465.981885174" watchObservedRunningTime="2026-10-31 01:00:05.761916496 +0000 UTC m=+2465.983952038"
Oct 31 01:00:05.805849 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:05.805818    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:06.395052 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:06.395020    4327 scope.go:122] "RemoveContainer" containerID="ebd0755e1ffe5f26b978139b6edd66e49ada20150ebd29b1a4cb104f905845a6"
Oct 31 01:00:06.395501 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:06.395471    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890140-k72t7_odh-ai-gateway-infra(b118e1e7-3dd9-4107-b740-9032c60dee40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7" podUID="b118e1e7-3dd9-4107-b740-9032c60dee40"
Oct 31 01:00:06.396996 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:06.396972    4327 scope.go:122] "RemoveContainer" containerID="b812e9c53991ed4940d9f843b4344eacb0f7284314a126576c42b8bc9575606b"
Oct 31 01:00:06.397425 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:06.397396    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn_odh-ai-gateway-infra(c4ee6cb0-2737-426b-9f00-4d9f550c8164)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn" podUID="c4ee6cb0-2737-426b-9f00-4d9f550c8164"
Oct 31 01:00:06.398966 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:06.398949    4327 scope.go:122] "RemoveContainer" containerID="3af96efdce87e5eea99bd030c8317493fb03d07a9828ce2a53772849c0d92b42"
Oct 31 01:00:06.399311 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:06.399283    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t_odh-ai-gateway-infra(7f085592-c63f-433e-a77f-0b0c20746fe2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t" podUID="7f085592-c63f-433e-a77f-0b0c20746fe2"
Oct 31 01:00:06.401210 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:06.401186    4327 scope.go:122] "RemoveContainer" containerID="61c9a8293ae387d46c835ebe3e6b03b0891a4bd225da234632b8ea0e62549a5d"
Oct 31 01:00:06.401505 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:06.401488    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt_odh-ai-gateway-infra(f45f589e-69d1-49fd-b27d-a15d1f886235)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt" podUID="f45f589e-69d1-49fd-b27d-a15d1f886235"
Oct 31 01:00:06.403145 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:06.403125    4327 scope.go:122] "RemoveContainer" containerID="5a3c3d924779efce8509d3d576c9456586a4adb2fd7d9afa91879f03b25eb75b"
Oct 31 01:00:06.403451 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:06.403429    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq_odh-ai-gateway-infra(11188116-a59b-48f1-9f7d-85d09032455e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq" podUID="11188116-a59b-48f1-9f7d-85d09032455e"
Oct 31 01:00:06.404945 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:06.404926    4327 scope.go:122] "RemoveContainer" containerID="0815f5e78b90ea10782475d318e3c6f4a5e7e4faf21c988cb3aa7474fca50e07"
Oct 31 01:00:06.405236 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:06.405213    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm_odh-ai-gateway-infra(8b6d9a4f-7b06-42f1-8e32-720489b7ba93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm" podUID="8b6d9a4f-7b06-42f1-8e32-720489b7ba93"
Oct 31 01:00:06.406559 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:06.406541    4327 scope.go:122] "RemoveContainer" containerID="958f370c31a0097a783153a04c95c10245e1971ac94b868ef40ce480d8b64d4c"
Oct 31 01:00:06.406894 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:06.406872    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8_odh-ai-gateway-infra(d352532d-5b9b-4713-a13c-32cd6d2c7716)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8" podUID="d352532d-5b9b-4713-a13c-32cd6d2c7716"
Oct 31 01:00:06.808145 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:06.808056    4327 patch_prober.go:29] interesting pod/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 31 01:00:06.808145 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:06.808110    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65" podUID="7a206da9-cfc0-41d4-888c-1123aab28e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 31 01:00:07.805971 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:07.805935    4327 patch_prober.go:29] interesting pod/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 31 01:00:07.806170 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:07.805989    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65" podUID="7a206da9-cfc0-41d4-888c-1123aab28e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 31 01:00:08.806110 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:08.806075    4327 patch_prober.go:29] interesting pod/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 31 01:00:08.806490 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:08.806132    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65" podUID="7a206da9-cfc0-41d4-888c-1123aab28e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 31 01:00:09.806319 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:09.806284    4327 patch_prober.go:29] interesting pod/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 31 01:00:09.806739 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:09.806338    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65" podUID="7a206da9-cfc0-41d4-888c-1123aab28e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 31 01:00:10.805858 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:10.805825    4327 patch_prober.go:29] interesting pod/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 31 01:00:10.806051 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:10.805879    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65" podUID="7a206da9-cfc0-41d4-888c-1123aab28e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 31 01:00:11.976367 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:11.976267    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:11.976876 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:11.976784    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:11.978992 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:11.978969    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:00:14.914222 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:14.914191    4327 scope.go:122] "RemoveContainer" containerID="61c9a8293ae387d46c835ebe3e6b03b0891a4bd225da234632b8ea0e62549a5d"
Oct 31 01:00:15.295401 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:15.295301    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:00:15.508416 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:15.508381    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt" event={"ID":"f45f589e-69d1-49fd-b27d-a15d1f886235","Type":"ContainerStarted","Data":"efc103c11b25afb701cb8bf94b60c4b663da954d23ad957b289f0e75d9fdd653"}
Oct 31 01:00:15.914163 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:15.914128    4327 scope.go:122] "RemoveContainer" containerID="ebd0755e1ffe5f26b978139b6edd66e49ada20150ebd29b1a4cb104f905845a6"
Oct 31 01:00:15.914557 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:15.914538    4327 scope.go:122] "RemoveContainer" containerID="998b2ec9486e13c9b45903d76962c562c3ee8d9cc5702164ff8b76ee72306bfe"
Oct 31 01:00:15.914913 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:15.914673    4327 scope.go:122] "RemoveContainer" containerID="0815f5e78b90ea10782475d318e3c6f4a5e7e4faf21c988cb3aa7474fca50e07"
Oct 31 01:00:16.519547 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:16.519505    4327 generic.go:356] "Generic (PLEG): container finished" podID="f45f589e-69d1-49fd-b27d-a15d1f886235" containerID="efc103c11b25afb701cb8bf94b60c4b663da954d23ad957b289f0e75d9fdd653" exitCode=60
Oct 31 01:00:16.519733 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:16.519544    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt" event={"ID":"f45f589e-69d1-49fd-b27d-a15d1f886235","Type":"ContainerDied","Data":"efc103c11b25afb701cb8bf94b60c4b663da954d23ad957b289f0e75d9fdd653"}
Oct 31 01:00:16.519733 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:16.519622    4327 scope.go:122] "RemoveContainer" containerID="61c9a8293ae387d46c835ebe3e6b03b0891a4bd225da234632b8ea0e62549a5d"
Oct 31 01:00:16.520190 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:16.520169    4327 scope.go:122] "RemoveContainer" containerID="efc103c11b25afb701cb8bf94b60c4b663da954d23ad957b289f0e75d9fdd653"
Oct 31 01:00:16.520796 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:16.520651    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt_odh-ai-gateway-infra(f45f589e-69d1-49fd-b27d-a15d1f886235)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt" podUID="f45f589e-69d1-49fd-b27d-a15d1f886235"
Oct 31 01:00:16.565031 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:16.564999    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt"]
Oct 31 01:00:16.914227 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:16.914129    4327 scope.go:122] "RemoveContainer" containerID="5a3c3d924779efce8509d3d576c9456586a4adb2fd7d9afa91879f03b25eb75b"
Oct 31 01:00:17.529955 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:17.529863    4327 generic.go:356] "Generic (PLEG): container finished" podID="b118e1e7-3dd9-4107-b740-9032c60dee40" containerID="528c71ebab5957bb43dfc74e9a1391432cbde42ee3c3be00f5ca947b408dffd5" exitCode=60
Oct 31 01:00:17.529955 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:17.529934    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7" event={"ID":"b118e1e7-3dd9-4107-b740-9032c60dee40","Type":"ContainerDied","Data":"528c71ebab5957bb43dfc74e9a1391432cbde42ee3c3be00f5ca947b408dffd5"}
Oct 31 01:00:17.530424 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:17.529998    4327 scope.go:122] "RemoveContainer" containerID="ebd0755e1ffe5f26b978139b6edd66e49ada20150ebd29b1a4cb104f905845a6"
Oct 31 01:00:17.530523 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:17.530502    4327 scope.go:122] "RemoveContainer" containerID="528c71ebab5957bb43dfc74e9a1391432cbde42ee3c3be00f5ca947b408dffd5"
Oct 31 01:00:17.531053 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:17.530987    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890140-k72t7_odh-ai-gateway-infra(b118e1e7-3dd9-4107-b740-9032c60dee40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7" podUID="b118e1e7-3dd9-4107-b740-9032c60dee40"
Oct 31 01:00:17.534296 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:17.534270    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq" event={"ID":"11188116-a59b-48f1-9f7d-85d09032455e","Type":"ContainerStarted","Data":"fd93a83f737658e746b18087ba0d98eeb2637a8fd046b744b3e30e32dc618403"}
Oct 31 01:00:17.536493 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:17.536471    4327 generic.go:356] "Generic (PLEG): container finished" podID="8b6d9a4f-7b06-42f1-8e32-720489b7ba93" containerID="101033b8989de92136927f45c242ec22ef412ba94edc0f4c1a473ff8014778f0" exitCode=60
Oct 31 01:00:17.536592 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:17.536544    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm" event={"ID":"8b6d9a4f-7b06-42f1-8e32-720489b7ba93","Type":"ContainerDied","Data":"101033b8989de92136927f45c242ec22ef412ba94edc0f4c1a473ff8014778f0"}
Oct 31 01:00:17.537008 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:17.536991    4327 scope.go:122] "RemoveContainer" containerID="101033b8989de92136927f45c242ec22ef412ba94edc0f4c1a473ff8014778f0"
Oct 31 01:00:17.537418 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:17.537390    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm_odh-ai-gateway-infra(8b6d9a4f-7b06-42f1-8e32-720489b7ba93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm" podUID="8b6d9a4f-7b06-42f1-8e32-720489b7ba93"
Oct 31 01:00:17.538888 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:17.538868    4327 generic.go:356] "Generic (PLEG): container finished" podID="13752887-b28f-49f8-855d-2eed4657dc61" containerID="7359486088e59a7fcb83b7fb7c955a3d4d41523ba24285bcdc85607c09c7f21b" exitCode=60
Oct 31 01:00:17.538986 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:17.538893    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7" event={"ID":"13752887-b28f-49f8-855d-2eed4657dc61","Type":"ContainerDied","Data":"7359486088e59a7fcb83b7fb7c955a3d4d41523ba24285bcdc85607c09c7f21b"}
Oct 31 01:00:17.539232 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:17.539215    4327 scope.go:122] "RemoveContainer" containerID="7359486088e59a7fcb83b7fb7c955a3d4d41523ba24285bcdc85607c09c7f21b"
Oct 31 01:00:17.539520 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:17.539504    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7_odh-ai-gateway-infra(13752887-b28f-49f8-855d-2eed4657dc61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7" podUID="13752887-b28f-49f8-855d-2eed4657dc61"
Oct 31 01:00:17.545323 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:17.545302    4327 scope.go:122] "RemoveContainer" containerID="0815f5e78b90ea10782475d318e3c6f4a5e7e4faf21c988cb3aa7474fca50e07"
Oct 31 01:00:17.560791 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:17.560762    4327 scope.go:122] "RemoveContainer" containerID="998b2ec9486e13c9b45903d76962c562c3ee8d9cc5702164ff8b76ee72306bfe"
Oct 31 01:00:17.723425 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:17.723398    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt"
Oct 31 01:00:17.771325 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:17.771298    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f45f589e-69d1-49fd-b27d-a15d1f886235-kube-api-access-8rndx\" (UniqueName: \"kubernetes.io/projected/f45f589e-69d1-49fd-b27d-a15d1f886235-kube-api-access-8rndx\") pod \"f45f589e-69d1-49fd-b27d-a15d1f886235\" (UID: \"f45f589e-69d1-49fd-b27d-a15d1f886235\") "
Oct 31 01:00:17.771548 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:17.771536    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f45f589e-69d1-49fd-b27d-a15d1f886235-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45f589e-69d1-49fd-b27d-a15d1f886235-maas-api-service-ca\") pod \"f45f589e-69d1-49fd-b27d-a15d1f886235\" (UID: \"f45f589e-69d1-49fd-b27d-a15d1f886235\") "
Oct 31 01:00:17.771921 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:17.771892    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f45f589e-69d1-49fd-b27d-a15d1f886235-maas-api-service-ca" pod "f45f589e-69d1-49fd-b27d-a15d1f886235" (UID: "f45f589e-69d1-49fd-b27d-a15d1f886235"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:00:17.772217 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:17.772200    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45f589e-69d1-49fd-b27d-a15d1f886235-maas-api-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:17.773457 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:17.773432    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45f589e-69d1-49fd-b27d-a15d1f886235-kube-api-access-8rndx" pod "f45f589e-69d1-49fd-b27d-a15d1f886235" (UID: "f45f589e-69d1-49fd-b27d-a15d1f886235"). InnerVolumeSpecName "kube-api-access-8rndx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:00:17.874186 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:17.874149    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rndx\" (UniqueName: \"kubernetes.io/projected/f45f589e-69d1-49fd-b27d-a15d1f886235-kube-api-access-8rndx\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:17.914904 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:17.914865    4327 scope.go:122] "RemoveContainer" containerID="958f370c31a0097a783153a04c95c10245e1971ac94b868ef40ce480d8b64d4c"
Oct 31 01:00:17.915275 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:17.915248    4327 scope.go:122] "RemoveContainer" containerID="b812e9c53991ed4940d9f843b4344eacb0f7284314a126576c42b8bc9575606b"
Oct 31 01:00:18.622704 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:18.618875    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt" event={"ID":"f45f589e-69d1-49fd-b27d-a15d1f886235","Type":"ContainerDied","Data":"8d8f099eac1c117bb2f335b7568f81d9fe50b851ea417422eea2847fa96e72a0"}
Oct 31 01:00:18.622704 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:18.618952    4327 scope.go:122] "RemoveContainer" containerID="efc103c11b25afb701cb8bf94b60c4b663da954d23ad957b289f0e75d9fdd653"
Oct 31 01:00:18.622704 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:18.619062    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt"
Oct 31 01:00:18.645205 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:18.638377    4327 generic.go:356] "Generic (PLEG): container finished" podID="11188116-a59b-48f1-9f7d-85d09032455e" containerID="fd93a83f737658e746b18087ba0d98eeb2637a8fd046b744b3e30e32dc618403" exitCode=60
Oct 31 01:00:18.645205 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:18.638513    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq" event={"ID":"11188116-a59b-48f1-9f7d-85d09032455e","Type":"ContainerDied","Data":"fd93a83f737658e746b18087ba0d98eeb2637a8fd046b744b3e30e32dc618403"}
Oct 31 01:00:18.645205 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:18.639160    4327 scope.go:122] "RemoveContainer" containerID="fd93a83f737658e746b18087ba0d98eeb2637a8fd046b744b3e30e32dc618403"
Oct 31 01:00:18.645205 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:18.639913    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq_odh-ai-gateway-infra(11188116-a59b-48f1-9f7d-85d09032455e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq" podUID="11188116-a59b-48f1-9f7d-85d09032455e"
Oct 31 01:00:18.651259 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:18.651221    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7"]
Oct 31 01:00:18.663166 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:18.663130    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7"]
Oct 31 01:00:18.667057 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:18.667030    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8" event={"ID":"d352532d-5b9b-4713-a13c-32cd6d2c7716","Type":"ContainerStarted","Data":"af6a1f7a9d68d856ebf2fd614ca59d41e7aa0207f8e0211d48e1202d0115b20e"}
Oct 31 01:00:18.672720 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:18.672657    4327 scope.go:122] "RemoveContainer" containerID="5a3c3d924779efce8509d3d576c9456586a4adb2fd7d9afa91879f03b25eb75b"
Oct 31 01:00:18.678275 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:18.677665    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn" event={"ID":"c4ee6cb0-2737-426b-9f00-4d9f550c8164","Type":"ContainerStarted","Data":"f3df31e970e5bb75b7c4e67ca2f7e96ca66e468e2bdff972609e20a047d62dee"}
Oct 31 01:00:18.711876 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:18.711834    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt"]
Oct 31 01:00:18.728955 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:18.728925    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-01a117-29890140-nbtgt"]
Oct 31 01:00:18.751616 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:18.751585    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq"]
Oct 31 01:00:18.816071 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:18.816044    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm"]
Oct 31 01:00:18.915727 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:18.914075    4327 scope.go:122] "RemoveContainer" containerID="3af96efdce87e5eea99bd030c8317493fb03d07a9828ce2a53772849c0d92b42"
Oct 31 01:00:18.954877 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:18.954796    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7"
Oct 31 01:00:19.003567 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.003529    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b118e1e7-3dd9-4107-b740-9032c60dee40-kube-api-access-4htbn\" (UniqueName: \"kubernetes.io/projected/b118e1e7-3dd9-4107-b740-9032c60dee40-kube-api-access-4htbn\") pod \"b118e1e7-3dd9-4107-b740-9032c60dee40\" (UID: \"b118e1e7-3dd9-4107-b740-9032c60dee40\") "
Oct 31 01:00:19.003849 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.003823    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b118e1e7-3dd9-4107-b740-9032c60dee40-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b118e1e7-3dd9-4107-b740-9032c60dee40-maas-api-service-ca\") pod \"b118e1e7-3dd9-4107-b740-9032c60dee40\" (UID: \"b118e1e7-3dd9-4107-b740-9032c60dee40\") "
Oct 31 01:00:19.004287 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.004253    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b118e1e7-3dd9-4107-b740-9032c60dee40-maas-api-service-ca" pod "b118e1e7-3dd9-4107-b740-9032c60dee40" (UID: "b118e1e7-3dd9-4107-b740-9032c60dee40"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:00:19.008183 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.008107    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b118e1e7-3dd9-4107-b740-9032c60dee40-maas-api-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:19.008183 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.008116    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b118e1e7-3dd9-4107-b740-9032c60dee40-kube-api-access-4htbn" pod "b118e1e7-3dd9-4107-b740-9032c60dee40" (UID: "b118e1e7-3dd9-4107-b740-9032c60dee40"). InnerVolumeSpecName "kube-api-access-4htbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:00:19.110114 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.110078    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4htbn\" (UniqueName: \"kubernetes.io/projected/b118e1e7-3dd9-4107-b740-9032c60dee40-kube-api-access-4htbn\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:19.119520 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.119488    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7"
Oct 31 01:00:19.127822 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.127797    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm"
Oct 31 01:00:19.211321 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.211239    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13752887-b28f-49f8-855d-2eed4657dc61-kube-api-access-6zwsk\" (UniqueName: \"kubernetes.io/projected/13752887-b28f-49f8-855d-2eed4657dc61-kube-api-access-6zwsk\") pod \"13752887-b28f-49f8-855d-2eed4657dc61\" (UID: \"13752887-b28f-49f8-855d-2eed4657dc61\") "
Oct 31 01:00:19.211459 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.211350    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b6d9a4f-7b06-42f1-8e32-720489b7ba93-kube-api-access-mpr94\" (UniqueName: \"kubernetes.io/projected/8b6d9a4f-7b06-42f1-8e32-720489b7ba93-kube-api-access-mpr94\") pod \"8b6d9a4f-7b06-42f1-8e32-720489b7ba93\" (UID: \"8b6d9a4f-7b06-42f1-8e32-720489b7ba93\") "
Oct 31 01:00:19.211459 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.211419    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13752887-b28f-49f8-855d-2eed4657dc61-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13752887-b28f-49f8-855d-2eed4657dc61-maas-api-service-ca\") pod \"13752887-b28f-49f8-855d-2eed4657dc61\" (UID: \"13752887-b28f-49f8-855d-2eed4657dc61\") "
Oct 31 01:00:19.211575 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.211465    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b6d9a4f-7b06-42f1-8e32-720489b7ba93-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b6d9a4f-7b06-42f1-8e32-720489b7ba93-maas-api-service-ca\") pod \"8b6d9a4f-7b06-42f1-8e32-720489b7ba93\" (UID: \"8b6d9a4f-7b06-42f1-8e32-720489b7ba93\") "
Oct 31 01:00:19.211851 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.211822    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b6d9a4f-7b06-42f1-8e32-720489b7ba93-maas-api-service-ca" pod "8b6d9a4f-7b06-42f1-8e32-720489b7ba93" (UID: "8b6d9a4f-7b06-42f1-8e32-720489b7ba93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:00:19.211851 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.211837    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13752887-b28f-49f8-855d-2eed4657dc61-maas-api-service-ca" pod "13752887-b28f-49f8-855d-2eed4657dc61" (UID: "13752887-b28f-49f8-855d-2eed4657dc61"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:00:19.212353 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.212327    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13752887-b28f-49f8-855d-2eed4657dc61-maas-api-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:19.212434 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.212358    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b6d9a4f-7b06-42f1-8e32-720489b7ba93-maas-api-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:19.221219 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.221187    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13752887-b28f-49f8-855d-2eed4657dc61-kube-api-access-6zwsk" pod "13752887-b28f-49f8-855d-2eed4657dc61" (UID: "13752887-b28f-49f8-855d-2eed4657dc61"). InnerVolumeSpecName "kube-api-access-6zwsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:00:19.224855 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.224825    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b6d9a4f-7b06-42f1-8e32-720489b7ba93-kube-api-access-mpr94" pod "8b6d9a4f-7b06-42f1-8e32-720489b7ba93" (UID: "8b6d9a4f-7b06-42f1-8e32-720489b7ba93"). InnerVolumeSpecName "kube-api-access-mpr94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:00:19.314156 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.314123    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zwsk\" (UniqueName: \"kubernetes.io/projected/13752887-b28f-49f8-855d-2eed4657dc61-kube-api-access-6zwsk\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:19.314156 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.314155    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpr94\" (UniqueName: \"kubernetes.io/projected/8b6d9a4f-7b06-42f1-8e32-720489b7ba93-kube-api-access-mpr94\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:19.687436 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.687400    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7"
Oct 31 01:00:19.687436 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.687412    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7" event={"ID":"b118e1e7-3dd9-4107-b740-9032c60dee40","Type":"ContainerDied","Data":"209d7bc2ac64cc962c365a0e9816338e4164e8dc66c03af8041a492614f87563"}
Oct 31 01:00:19.688038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.687476    4327 scope.go:122] "RemoveContainer" containerID="528c71ebab5957bb43dfc74e9a1391432cbde42ee3c3be00f5ca947b408dffd5"
Oct 31 01:00:19.689883 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.689862    4327 generic.go:356] "Generic (PLEG): container finished" podID="c4ee6cb0-2737-426b-9f00-4d9f550c8164" containerID="f3df31e970e5bb75b7c4e67ca2f7e96ca66e468e2bdff972609e20a047d62dee" exitCode=60
Oct 31 01:00:19.689983 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.689940    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn" event={"ID":"c4ee6cb0-2737-426b-9f00-4d9f550c8164","Type":"ContainerDied","Data":"f3df31e970e5bb75b7c4e67ca2f7e96ca66e468e2bdff972609e20a047d62dee"}
Oct 31 01:00:19.690466 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.690447    4327 scope.go:122] "RemoveContainer" containerID="f3df31e970e5bb75b7c4e67ca2f7e96ca66e468e2bdff972609e20a047d62dee"
Oct 31 01:00:19.690892 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:19.690868    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn_odh-ai-gateway-infra(c4ee6cb0-2737-426b-9f00-4d9f550c8164)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn" podUID="c4ee6cb0-2737-426b-9f00-4d9f550c8164"
Oct 31 01:00:19.692350 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.692325    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t" event={"ID":"7f085592-c63f-433e-a77f-0b0c20746fe2","Type":"ContainerStarted","Data":"90d2fce6840e16ec64f9baf4cff5294489d6b2e02f29832746a3ef0fd52c480c"}
Oct 31 01:00:19.696828 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.696791    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm"
Oct 31 01:00:19.696828 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.696810    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm" event={"ID":"8b6d9a4f-7b06-42f1-8e32-720489b7ba93","Type":"ContainerDied","Data":"5e5805e2d055163a00f5a03cd97818fa3acf3ea3f7ca6d4b73ab6505789010b2"}
Oct 31 01:00:19.698621 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.698606    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7"
Oct 31 01:00:19.698743 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.698642    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7" event={"ID":"13752887-b28f-49f8-855d-2eed4657dc61","Type":"ContainerDied","Data":"7f6f75a511fb25fd16231bf9f5253f3fedc6eb5c8b99440538e348cd980188e1"}
Oct 31 01:00:19.700836 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.700809    4327 generic.go:356] "Generic (PLEG): container finished" podID="d352532d-5b9b-4713-a13c-32cd6d2c7716" containerID="af6a1f7a9d68d856ebf2fd614ca59d41e7aa0207f8e0211d48e1202d0115b20e" exitCode=60
Oct 31 01:00:19.700944 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.700887    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8" event={"ID":"d352532d-5b9b-4713-a13c-32cd6d2c7716","Type":"ContainerDied","Data":"af6a1f7a9d68d856ebf2fd614ca59d41e7aa0207f8e0211d48e1202d0115b20e"}
Oct 31 01:00:19.701382 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.701363    4327 scope.go:122] "RemoveContainer" containerID="af6a1f7a9d68d856ebf2fd614ca59d41e7aa0207f8e0211d48e1202d0115b20e"
Oct 31 01:00:19.701608 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.701591    4327 scope.go:122] "RemoveContainer" containerID="b812e9c53991ed4940d9f843b4344eacb0f7284314a126576c42b8bc9575606b"
Oct 31 01:00:19.701662 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:19.701632    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8_odh-ai-gateway-infra(d352532d-5b9b-4713-a13c-32cd6d2c7716)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8" podUID="d352532d-5b9b-4713-a13c-32cd6d2c7716"
Oct 31 01:00:19.717735 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.717706    4327 scope.go:122] "RemoveContainer" containerID="101033b8989de92136927f45c242ec22ef412ba94edc0f4c1a473ff8014778f0"
Oct 31 01:00:19.728581 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.728560    4327 scope.go:122] "RemoveContainer" containerID="7359486088e59a7fcb83b7fb7c955a3d4d41523ba24285bcdc85607c09c7f21b"
Oct 31 01:00:19.741953 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.741927    4327 scope.go:122] "RemoveContainer" containerID="958f370c31a0097a783153a04c95c10245e1971ac94b868ef40ce480d8b64d4c"
Oct 31 01:00:19.876597 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.876517    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7"]
Oct 31 01:00:19.891311 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.891276    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890140-k72t7"]
Oct 31 01:00:19.894757 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.894735    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq"
Oct 31 01:00:19.923159 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.923120    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11188116-a59b-48f1-9f7d-85d09032455e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11188116-a59b-48f1-9f7d-85d09032455e-maas-api-service-ca\") pod \"11188116-a59b-48f1-9f7d-85d09032455e\" (UID: \"11188116-a59b-48f1-9f7d-85d09032455e\") "
Oct 31 01:00:19.923315 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.923193    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11188116-a59b-48f1-9f7d-85d09032455e-kube-api-access-cvzjh\" (UniqueName: \"kubernetes.io/projected/11188116-a59b-48f1-9f7d-85d09032455e-kube-api-access-cvzjh\") pod \"11188116-a59b-48f1-9f7d-85d09032455e\" (UID: \"11188116-a59b-48f1-9f7d-85d09032455e\") "
Oct 31 01:00:19.923532 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.923505    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11188116-a59b-48f1-9f7d-85d09032455e-maas-api-service-ca" pod "11188116-a59b-48f1-9f7d-85d09032455e" (UID: "11188116-a59b-48f1-9f7d-85d09032455e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:00:19.924180 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.924152    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11188116-a59b-48f1-9f7d-85d09032455e-maas-api-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:19.925570 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.925545    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b118e1e7-3dd9-4107-b740-9032c60dee40" path="/var/lib/kubelet/pods/b118e1e7-3dd9-4107-b740-9032c60dee40/volumes"
Oct 31 01:00:19.925789 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.925645    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11188116-a59b-48f1-9f7d-85d09032455e-kube-api-access-cvzjh" pod "11188116-a59b-48f1-9f7d-85d09032455e" (UID: "11188116-a59b-48f1-9f7d-85d09032455e"). InnerVolumeSpecName "kube-api-access-cvzjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:00:19.926447 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.926426    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f45f589e-69d1-49fd-b27d-a15d1f886235" path="/var/lib/kubelet/pods/f45f589e-69d1-49fd-b27d-a15d1f886235/volumes"
Oct 31 01:00:19.964541 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:19.964510    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8"]
Oct 31 01:00:20.025922 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:20.025885    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvzjh\" (UniqueName: \"kubernetes.io/projected/11188116-a59b-48f1-9f7d-85d09032455e-kube-api-access-cvzjh\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:20.035825 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:20.035793    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7"]
Oct 31 01:00:20.041895 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:20.041863    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890140-j9jm7"]
Oct 31 01:00:20.083743 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:20.083296    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn"]
Oct 31 01:00:20.145069 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:20.144937    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm"]
Oct 31 01:00:20.152943 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:20.152906    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890140-wrdqm"]
Oct 31 01:00:20.725836 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:20.725798    4327 generic.go:356] "Generic (PLEG): container finished" podID="7f085592-c63f-433e-a77f-0b0c20746fe2" containerID="90d2fce6840e16ec64f9baf4cff5294489d6b2e02f29832746a3ef0fd52c480c" exitCode=60
Oct 31 01:00:20.726362 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:20.725872    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t" event={"ID":"7f085592-c63f-433e-a77f-0b0c20746fe2","Type":"ContainerDied","Data":"90d2fce6840e16ec64f9baf4cff5294489d6b2e02f29832746a3ef0fd52c480c"}
Oct 31 01:00:20.726362 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:20.725989    4327 scope.go:122] "RemoveContainer" containerID="3af96efdce87e5eea99bd030c8317493fb03d07a9828ce2a53772849c0d92b42"
Oct 31 01:00:20.726506 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:20.726490    4327 scope.go:122] "RemoveContainer" containerID="90d2fce6840e16ec64f9baf4cff5294489d6b2e02f29832746a3ef0fd52c480c"
Oct 31 01:00:20.726992 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:20.726957    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t_odh-ai-gateway-infra(7f085592-c63f-433e-a77f-0b0c20746fe2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t" podUID="7f085592-c63f-433e-a77f-0b0c20746fe2"
Oct 31 01:00:20.727656 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:20.727633    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq"
Oct 31 01:00:20.727786 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:20.727629    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq" event={"ID":"11188116-a59b-48f1-9f7d-85d09032455e","Type":"ContainerDied","Data":"1f6740b6efb80f8d219a129a64ef90600ffeb092548b3a251af4dc268f06657d"}
Oct 31 01:00:20.743159 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:20.743130    4327 scope.go:122] "RemoveContainer" containerID="fd93a83f737658e746b18087ba0d98eeb2637a8fd046b744b3e30e32dc618403"
Oct 31 01:00:20.831091 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:20.831059    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq"]
Oct 31 01:00:20.846152 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:20.846126    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d5621-29890140-skjqq"]
Oct 31 01:00:20.853399 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:20.853370    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t"]
Oct 31 01:00:20.938858 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:20.938827    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn"
Oct 31 01:00:20.994697 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:20.994609    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8"
Oct 31 01:00:21.046950 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:21.046911    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ee6cb0-2737-426b-9f00-4d9f550c8164-kube-api-access-cnljm\" (UniqueName: \"kubernetes.io/projected/c4ee6cb0-2737-426b-9f00-4d9f550c8164-kube-api-access-cnljm\") pod \"c4ee6cb0-2737-426b-9f00-4d9f550c8164\" (UID: \"c4ee6cb0-2737-426b-9f00-4d9f550c8164\") "
Oct 31 01:00:21.047384 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:21.047344    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d352532d-5b9b-4713-a13c-32cd6d2c7716-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d352532d-5b9b-4713-a13c-32cd6d2c7716-maas-api-service-ca\") pod \"d352532d-5b9b-4713-a13c-32cd6d2c7716\" (UID: \"d352532d-5b9b-4713-a13c-32cd6d2c7716\") "
Oct 31 01:00:21.047625 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:21.047391    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4ee6cb0-2737-426b-9f00-4d9f550c8164-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ee6cb0-2737-426b-9f00-4d9f550c8164-maas-api-service-ca\") pod \"c4ee6cb0-2737-426b-9f00-4d9f550c8164\" (UID: \"c4ee6cb0-2737-426b-9f00-4d9f550c8164\") "
Oct 31 01:00:21.050313 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:21.050278    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d352532d-5b9b-4713-a13c-32cd6d2c7716-maas-api-service-ca" pod "d352532d-5b9b-4713-a13c-32cd6d2c7716" (UID: "d352532d-5b9b-4713-a13c-32cd6d2c7716"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:00:21.050507 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:21.050484    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d352532d-5b9b-4713-a13c-32cd6d2c7716-kube-api-access-ss9ph\" (UniqueName: \"kubernetes.io/projected/d352532d-5b9b-4713-a13c-32cd6d2c7716-kube-api-access-ss9ph\") pod \"d352532d-5b9b-4713-a13c-32cd6d2c7716\" (UID: \"d352532d-5b9b-4713-a13c-32cd6d2c7716\") "
Oct 31 01:00:21.050745 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:21.050716    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4ee6cb0-2737-426b-9f00-4d9f550c8164-maas-api-service-ca" pod "c4ee6cb0-2737-426b-9f00-4d9f550c8164" (UID: "c4ee6cb0-2737-426b-9f00-4d9f550c8164"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:00:21.050997 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:21.050960    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ee6cb0-2737-426b-9f00-4d9f550c8164-kube-api-access-cnljm" pod "c4ee6cb0-2737-426b-9f00-4d9f550c8164" (UID: "c4ee6cb0-2737-426b-9f00-4d9f550c8164"). InnerVolumeSpecName "kube-api-access-cnljm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:00:21.053031 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:21.052897    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnljm\" (UniqueName: \"kubernetes.io/projected/c4ee6cb0-2737-426b-9f00-4d9f550c8164-kube-api-access-cnljm\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:21.053031 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:21.052926    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d352532d-5b9b-4713-a13c-32cd6d2c7716-maas-api-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:21.053031 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:21.052943    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ee6cb0-2737-426b-9f00-4d9f550c8164-maas-api-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:21.054007 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:21.053569    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d352532d-5b9b-4713-a13c-32cd6d2c7716-kube-api-access-ss9ph" pod "d352532d-5b9b-4713-a13c-32cd6d2c7716" (UID: "d352532d-5b9b-4713-a13c-32cd6d2c7716"). InnerVolumeSpecName "kube-api-access-ss9ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:00:21.155129 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:21.155092    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss9ph\" (UniqueName: \"kubernetes.io/projected/d352532d-5b9b-4713-a13c-32cd6d2c7716-kube-api-access-ss9ph\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:21.645427 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:21.645385    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"]
Oct 31 01:00:21.645650 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:21.645623    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh" podUID="2a45ba96-dddd-43f5-9e50-0b08194799f8" containerName="istio-proxy" containerID="cri-o://5637bf6d1b2b20701bc575f8f659153d51766c4bef7227af3fa93210f9101723" gracePeriod=30
Oct 31 01:00:21.743031 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:21.742998    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn" event={"ID":"c4ee6cb0-2737-426b-9f00-4d9f550c8164","Type":"ContainerDied","Data":"fc500358463fd890b8b3bf071778492c46b0941b61b75bda8986664b9f74d23e"}
Oct 31 01:00:21.743487 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:21.743005    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn"
Oct 31 01:00:21.743487 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:21.743065    4327 scope.go:122] "RemoveContainer" containerID="f3df31e970e5bb75b7c4e67ca2f7e96ca66e468e2bdff972609e20a047d62dee"
Oct 31 01:00:21.770858 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:21.770824    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8" event={"ID":"d352532d-5b9b-4713-a13c-32cd6d2c7716","Type":"ContainerDied","Data":"6790918b4979ef31ebaf3f51243e5438403ed845fb72f65f233f53e1ba65ec89"}
Oct 31 01:00:21.771062 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:21.770942    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8"
Oct 31 01:00:21.778040 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:21.778022    4327 scope.go:122] "RemoveContainer" containerID="af6a1f7a9d68d856ebf2fd614ca59d41e7aa0207f8e0211d48e1202d0115b20e"
Oct 31 01:00:21.892093 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:21.892054    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn"]
Oct 31 01:00:21.925745 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:21.925669    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11188116-a59b-48f1-9f7d-85d09032455e" path="/var/lib/kubelet/pods/11188116-a59b-48f1-9f7d-85d09032455e/volumes"
Oct 31 01:00:21.926501 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:21.926480    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13752887-b28f-49f8-855d-2eed4657dc61" path="/var/lib/kubelet/pods/13752887-b28f-49f8-855d-2eed4657dc61/volumes"
Oct 31 01:00:21.927020 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:21.927004    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b6d9a4f-7b06-42f1-8e32-720489b7ba93" path="/var/lib/kubelet/pods/8b6d9a4f-7b06-42f1-8e32-720489b7ba93/volumes"
Oct 31 01:00:21.927779 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:21.927759    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9de7f5-29890140-nr8dn"]
Oct 31 01:00:21.960751 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:21.960724    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t"
Oct 31 01:00:22.002414 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:22.002381    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8"]
Oct 31 01:00:22.018020 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:22.017990    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890140-njnr8"]
Oct 31 01:00:22.082953 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:22.082918    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f085592-c63f-433e-a77f-0b0c20746fe2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f085592-c63f-433e-a77f-0b0c20746fe2-maas-api-service-ca\") pod \"7f085592-c63f-433e-a77f-0b0c20746fe2\" (UID: \"7f085592-c63f-433e-a77f-0b0c20746fe2\") "
Oct 31 01:00:22.083522 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:22.083375    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f085592-c63f-433e-a77f-0b0c20746fe2-maas-api-service-ca" pod "7f085592-c63f-433e-a77f-0b0c20746fe2" (UID: "7f085592-c63f-433e-a77f-0b0c20746fe2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:00:22.084744 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:22.084279    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f085592-c63f-433e-a77f-0b0c20746fe2-kube-api-access-tmkng\" (UniqueName: \"kubernetes.io/projected/7f085592-c63f-433e-a77f-0b0c20746fe2-kube-api-access-tmkng\") pod \"7f085592-c63f-433e-a77f-0b0c20746fe2\" (UID: \"7f085592-c63f-433e-a77f-0b0c20746fe2\") "
Oct 31 01:00:22.085384 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:22.085358    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f085592-c63f-433e-a77f-0b0c20746fe2-maas-api-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:22.087849 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:22.087810    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f085592-c63f-433e-a77f-0b0c20746fe2-kube-api-access-tmkng" pod "7f085592-c63f-433e-a77f-0b0c20746fe2" (UID: "7f085592-c63f-433e-a77f-0b0c20746fe2"). InnerVolumeSpecName "kube-api-access-tmkng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:00:22.187306 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:22.187116    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmkng\" (UniqueName: \"kubernetes.io/projected/7f085592-c63f-433e-a77f-0b0c20746fe2-kube-api-access-tmkng\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:22.782571 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:22.782534    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t"
Oct 31 01:00:22.783103 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:22.782539    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t" event={"ID":"7f085592-c63f-433e-a77f-0b0c20746fe2","Type":"ContainerDied","Data":"5c4c65b5492a1bf35b064c91519f917165d95548c109c58457655282d0554cc0"}
Oct 31 01:00:22.783103 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:22.782724    4327 scope.go:122] "RemoveContainer" containerID="90d2fce6840e16ec64f9baf4cff5294489d6b2e02f29832746a3ef0fd52c480c"
Oct 31 01:00:22.839048 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:22.839018    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t"]
Oct 31 01:00:22.843651 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:22.843624    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890140-wlh6t"]
Oct 31 01:00:23.216444 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:23.216409    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"]
Oct 31 01:00:23.925852 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:23.925816    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f085592-c63f-433e-a77f-0b0c20746fe2" path="/var/lib/kubelet/pods/7f085592-c63f-433e-a77f-0b0c20746fe2/volumes"
Oct 31 01:00:23.926454 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:23.926433    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4ee6cb0-2737-426b-9f00-4d9f550c8164" path="/var/lib/kubelet/pods/c4ee6cb0-2737-426b-9f00-4d9f550c8164/volumes"
Oct 31 01:00:23.927122 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:23.927100    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d352532d-5b9b-4713-a13c-32cd6d2c7716" path="/var/lib/kubelet/pods/d352532d-5b9b-4713-a13c-32cd6d2c7716/volumes"
Oct 31 01:00:25.194406 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:25.194370    4327 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:00:25.194824 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:25.194424    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh" podUID="2a45ba96-dddd-43f5-9e50-0b08194799f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:00:26.826767 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:26.826733    4327 generic.go:356] "Generic (PLEG): container finished" podID="2a45ba96-dddd-43f5-9e50-0b08194799f8" containerID="5637bf6d1b2b20701bc575f8f659153d51766c4bef7227af3fa93210f9101723" exitCode=0
Oct 31 01:00:26.827213 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:26.826807    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh" event={"ID":"2a45ba96-dddd-43f5-9e50-0b08194799f8","Type":"ContainerDied","Data":"5637bf6d1b2b20701bc575f8f659153d51766c4bef7227af3fa93210f9101723"}
Oct 31 01:00:26.946604 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:26.946578    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 01:00:27.031371 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.031334    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-workload-socket\") pod \"2a45ba96-dddd-43f5-9e50-0b08194799f8\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") "
Oct 31 01:00:27.031576 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.031389    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-workload-certs\") pod \"2a45ba96-dddd-43f5-9e50-0b08194799f8\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") "
Oct 31 01:00:27.031576 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.031423    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-data\") pod \"2a45ba96-dddd-43f5-9e50-0b08194799f8\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") "
Oct 31 01:00:27.031576 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.031470    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-envoy\") pod \"2a45ba96-dddd-43f5-9e50-0b08194799f8\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") "
Oct 31 01:00:27.031837 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.031627    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-credential-socket\") pod \"2a45ba96-dddd-43f5-9e50-0b08194799f8\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") "
Oct 31 01:00:27.031837 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.031668    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-podinfo\") pod \"2a45ba96-dddd-43f5-9e50-0b08194799f8\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") "
Oct 31 01:00:27.031837 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.031749    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a45ba96-dddd-43f5-9e50-0b08194799f8-kube-api-access-dtg4p\" (UniqueName: \"kubernetes.io/projected/2a45ba96-dddd-43f5-9e50-0b08194799f8-kube-api-access-dtg4p\") pod \"2a45ba96-dddd-43f5-9e50-0b08194799f8\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") "
Oct 31 01:00:27.031837 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.031815    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a45ba96-dddd-43f5-9e50-0b08194799f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a45ba96-dddd-43f5-9e50-0b08194799f8-istiod-ca-cert\") pod \"2a45ba96-dddd-43f5-9e50-0b08194799f8\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") "
Oct 31 01:00:27.032084 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.031890    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-token\" (UniqueName: \"kubernetes.io/projected/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-token\") pod \"2a45ba96-dddd-43f5-9e50-0b08194799f8\" (UID: \"2a45ba96-dddd-43f5-9e50-0b08194799f8\") "
Oct 31 01:00:27.034177 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.033670    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-credential-socket" pod "2a45ba96-dddd-43f5-9e50-0b08194799f8" (UID: "2a45ba96-dddd-43f5-9e50-0b08194799f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:27.034177 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.033928    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-data" pod "2a45ba96-dddd-43f5-9e50-0b08194799f8" (UID: "2a45ba96-dddd-43f5-9e50-0b08194799f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:27.034177 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.034125    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-workload-socket" pod "2a45ba96-dddd-43f5-9e50-0b08194799f8" (UID: "2a45ba96-dddd-43f5-9e50-0b08194799f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:27.034444 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.034234    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-workload-certs" pod "2a45ba96-dddd-43f5-9e50-0b08194799f8" (UID: "2a45ba96-dddd-43f5-9e50-0b08194799f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:27.034444 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.034405    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a45ba96-dddd-43f5-9e50-0b08194799f8-istiod-ca-cert" pod "2a45ba96-dddd-43f5-9e50-0b08194799f8" (UID: "2a45ba96-dddd-43f5-9e50-0b08194799f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:00:27.036283 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.036253    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-token" pod "2a45ba96-dddd-43f5-9e50-0b08194799f8" (UID: "2a45ba96-dddd-43f5-9e50-0b08194799f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:00:27.036473 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.036417    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-envoy" pod "2a45ba96-dddd-43f5-9e50-0b08194799f8" (UID: "2a45ba96-dddd-43f5-9e50-0b08194799f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:27.040437 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.040407    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-podinfo" pod "2a45ba96-dddd-43f5-9e50-0b08194799f8" (UID: "2a45ba96-dddd-43f5-9e50-0b08194799f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:00:27.040605 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.040586    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a45ba96-dddd-43f5-9e50-0b08194799f8-kube-api-access-dtg4p" pod "2a45ba96-dddd-43f5-9e50-0b08194799f8" (UID: "2a45ba96-dddd-43f5-9e50-0b08194799f8"). InnerVolumeSpecName "kube-api-access-dtg4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:00:27.136967 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.136924    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:27.136967 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.136967    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:27.137232 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.136983    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:27.137232 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.136998    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:27.137232 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.137018    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:27.137232 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.137041    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a45ba96-dddd-43f5-9e50-0b08194799f8-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:27.137232 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.137055    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a45ba96-dddd-43f5-9e50-0b08194799f8-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:27.137232 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.137068    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtg4p\" (UniqueName: \"kubernetes.io/projected/2a45ba96-dddd-43f5-9e50-0b08194799f8-kube-api-access-dtg4p\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:27.137232 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.137087    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a45ba96-dddd-43f5-9e50-0b08194799f8-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:27.837298 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.837269    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"
Oct 31 01:00:27.837298 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.837282    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh" event={"ID":"2a45ba96-dddd-43f5-9e50-0b08194799f8","Type":"ContainerDied","Data":"54d4e4e74317d54fad5fdc84f056b0d0218492e86b4867b3b530952fc2153ac7"}
Oct 31 01:00:27.837876 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.837383    4327 scope.go:122] "RemoveContainer" containerID="5637bf6d1b2b20701bc575f8f659153d51766c4bef7227af3fa93210f9101723"
Oct 31 01:00:27.983997 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:27.983949    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"]
Oct 31 01:00:28.063702 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:28.058079    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-dldvh"]
Oct 31 01:00:29.924124 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:29.924095    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a45ba96-dddd-43f5-9e50-0b08194799f8" path="/var/lib/kubelet/pods/2a45ba96-dddd-43f5-9e50-0b08194799f8/volumes"
Oct 31 01:00:34.007752 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:34.007713    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs"]
Oct 31 01:00:34.040333 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:34.040299    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz"]
Oct 31 01:00:38.239746 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:38.239670    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8" podUID="a175adcd-0c7a-4572-940c-96eca0ee58d5" containerName="main" containerID="cri-o://c2fe3a1404b72d3d5d21d27039238d846b03c8f1d1b77b416478d63d60df6dd3" gracePeriod=45
Oct 31 01:00:38.601463 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:38.601424    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"
Oct 31 01:00:38.722837 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:38.722802    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-dshm\") pod \"a175adcd-0c7a-4572-940c-96eca0ee58d5\" (UID: \"a175adcd-0c7a-4572-940c-96eca0ee58d5\") "
Oct 31 01:00:38.723023 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:38.722891    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a175adcd-0c7a-4572-940c-96eca0ee58d5-tls-certs\" (UniqueName: \"kubernetes.io/secret/a175adcd-0c7a-4572-940c-96eca0ee58d5-tls-certs\") pod \"a175adcd-0c7a-4572-940c-96eca0ee58d5\" (UID: \"a175adcd-0c7a-4572-940c-96eca0ee58d5\") "
Oct 31 01:00:38.723023 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:38.722937    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-home\" (UniqueName: \"kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-home\") pod \"a175adcd-0c7a-4572-940c-96eca0ee58d5\" (UID: \"a175adcd-0c7a-4572-940c-96eca0ee58d5\") "
Oct 31 01:00:38.723023 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:38.722960    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a175adcd-0c7a-4572-940c-96eca0ee58d5-kube-api-access-dp4gz\" (UniqueName: \"kubernetes.io/projected/a175adcd-0c7a-4572-940c-96eca0ee58d5-kube-api-access-dp4gz\") pod \"a175adcd-0c7a-4572-940c-96eca0ee58d5\" (UID: \"a175adcd-0c7a-4572-940c-96eca0ee58d5\") "
Oct 31 01:00:38.723205 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:38.723040    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-tmp-dir\") pod \"a175adcd-0c7a-4572-940c-96eca0ee58d5\" (UID: \"a175adcd-0c7a-4572-940c-96eca0ee58d5\") "
Oct 31 01:00:38.723205 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:38.723092    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-model-cache\") pod \"a175adcd-0c7a-4572-940c-96eca0ee58d5\" (UID: \"a175adcd-0c7a-4572-940c-96eca0ee58d5\") "
Oct 31 01:00:38.723315 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:38.723235    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-home" pod "a175adcd-0c7a-4572-940c-96eca0ee58d5" (UID: "a175adcd-0c7a-4572-940c-96eca0ee58d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:38.723421 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:38.723391    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-tmp-dir" pod "a175adcd-0c7a-4572-940c-96eca0ee58d5" (UID: "a175adcd-0c7a-4572-940c-96eca0ee58d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:38.723556 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:38.723397    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-model-cache" pod "a175adcd-0c7a-4572-940c-96eca0ee58d5" (UID: "a175adcd-0c7a-4572-940c-96eca0ee58d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:38.723909 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:38.723887    4327 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-home\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:38.723997 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:38.723934    4327 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-tmp-dir\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:38.723997 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:38.723950    4327 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-model-cache\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:38.725169 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:38.725142    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-dshm" pod "a175adcd-0c7a-4572-940c-96eca0ee58d5" (UID: "a175adcd-0c7a-4572-940c-96eca0ee58d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:38.725258 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:38.725204    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a175adcd-0c7a-4572-940c-96eca0ee58d5-tls-certs" pod "a175adcd-0c7a-4572-940c-96eca0ee58d5" (UID: "a175adcd-0c7a-4572-940c-96eca0ee58d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:00:38.725321 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:38.725253    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a175adcd-0c7a-4572-940c-96eca0ee58d5-kube-api-access-dp4gz" pod "a175adcd-0c7a-4572-940c-96eca0ee58d5" (UID: "a175adcd-0c7a-4572-940c-96eca0ee58d5"). InnerVolumeSpecName "kube-api-access-dp4gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:00:38.826092 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:38.825995    4327 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a175adcd-0c7a-4572-940c-96eca0ee58d5-dshm\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:38.826092 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:38.826027    4327 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a175adcd-0c7a-4572-940c-96eca0ee58d5-tls-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:38.826092 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:38.826043    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp4gz\" (UniqueName: \"kubernetes.io/projected/a175adcd-0c7a-4572-940c-96eca0ee58d5-kube-api-access-dp4gz\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:39.009014 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:39.008969    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs" podUID="a9369d90-6579-4f6a-befa-eb8a84e09bc0" containerName="payload-processing" containerID="cri-o://96f71e10b231416b3a68ce099d40f41e8eb80842a419478e38da90a444c87566" gracePeriod=25
Oct 31 01:00:39.040833 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:39.040797    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz" podUID="dccc262c-1c89-428b-a7a0-5d931327bf82" containerName="payload-pre-processing" containerID="cri-o://501f86f6c351e8129eb3ccaa001b5d72491fd4cb9ae2c59ba5f0a99233cc3406" gracePeriod=25
Oct 31 01:00:39.050861 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:39.050829    4327 generic.go:356] "Generic (PLEG): container finished" podID="a175adcd-0c7a-4572-940c-96eca0ee58d5" containerID="c2fe3a1404b72d3d5d21d27039238d846b03c8f1d1b77b416478d63d60df6dd3" exitCode=0
Oct 31 01:00:39.051039 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:39.050868    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"
Oct 31 01:00:39.051039 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:39.050909    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8" event={"ID":"a175adcd-0c7a-4572-940c-96eca0ee58d5","Type":"ContainerDied","Data":"c2fe3a1404b72d3d5d21d27039238d846b03c8f1d1b77b416478d63d60df6dd3"}
Oct 31 01:00:39.051039 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:39.050966    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8" event={"ID":"a175adcd-0c7a-4572-940c-96eca0ee58d5","Type":"ContainerDied","Data":"5a39d6acf70ccc76f63a5fa3547c80768edd3ec033970f40ba9f66f61365ce2b"}
Oct 31 01:00:39.051039 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:39.050991    4327 scope.go:122] "RemoveContainer" containerID="c2fe3a1404b72d3d5d21d27039238d846b03c8f1d1b77b416478d63d60df6dd3"
Oct 31 01:00:39.064752 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:39.064729    4327 scope.go:122] "RemoveContainer" containerID="c2fe3a1404b72d3d5d21d27039238d846b03c8f1d1b77b416478d63d60df6dd3"
Oct 31 01:00:39.065034 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:39.065012    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2fe3a1404b72d3d5d21d27039238d846b03c8f1d1b77b416478d63d60df6dd3\": container with ID starting with c2fe3a1404b72d3d5d21d27039238d846b03c8f1d1b77b416478d63d60df6dd3 not found: ID does not exist" containerID="c2fe3a1404b72d3d5d21d27039238d846b03c8f1d1b77b416478d63d60df6dd3"
Oct 31 01:00:39.065105 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:39.065045    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2fe3a1404b72d3d5d21d27039238d846b03c8f1d1b77b416478d63d60df6dd3"} err="failed to get container status \"c2fe3a1404b72d3d5d21d27039238d846b03c8f1d1b77b416478d63d60df6dd3\": rpc error: code = NotFound desc = could not find container \"c2fe3a1404b72d3d5d21d27039238d846b03c8f1d1b77b416478d63d60df6dd3\": container with ID starting with c2fe3a1404b72d3d5d21d27039238d846b03c8f1d1b77b416478d63d60df6dd3 not found: ID does not exist"
Oct 31 01:00:39.105738 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:39.105709    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"]
Oct 31 01:00:39.118416 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:39.118388    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c6d97d/test-model-c6d97d-kserve-7585856c57-x56f8"]
Oct 31 01:00:39.898054 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:39.898016    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz" podUID="dccc262c-1c89-428b-a7a0-5d931327bf82" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:00:39.898054 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:39.898049    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs" podUID="a9369d90-6579-4f6a-befa-eb8a84e09bc0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:00:39.925148 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:39.925117    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a175adcd-0c7a-4572-940c-96eca0ee58d5" path="/var/lib/kubelet/pods/a175adcd-0c7a-4572-940c-96eca0ee58d5/volumes"
Oct 31 01:00:39.970747 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:39.970647    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"]
Oct 31 01:00:45.602797 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:45.602755    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:00:49.898274 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:49.898231    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz" podUID="dccc262c-1c89-428b-a7a0-5d931327bf82" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:00:49.898843 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:49.898296    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs" podUID="a9369d90-6579-4f6a-befa-eb8a84e09bc0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:00:53.433059 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:53.433016    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"]
Oct 31 01:00:53.433969 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:53.433926    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht" podUID="e30c1c23-92c5-47b6-a250-3519dbfae94b" containerName="istio-proxy" containerID="cri-o://7a7aec5ea85a41c0f310a46441a132ed668b7b400d2679fc50f3cbd031e5cf49" gracePeriod=30
Oct 31 01:00:54.994404 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:54.994357    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf" podUID="734185a4-0ff7-4580-abb7-865dee21b8aa" containerName="main" containerID="cri-o://1186698632d7057827622a69c91745082dee4d6767b902c74364eae18ed3a872" gracePeriod=45
Oct 31 01:00:55.217550 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:55.217503    4327 generic.go:356] "Generic (PLEG): container finished" podID="734185a4-0ff7-4580-abb7-865dee21b8aa" containerID="1186698632d7057827622a69c91745082dee4d6767b902c74364eae18ed3a872" exitCode=0
Oct 31 01:00:55.217765 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:55.217628    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf" event={"ID":"734185a4-0ff7-4580-abb7-865dee21b8aa","Type":"ContainerDied","Data":"1186698632d7057827622a69c91745082dee4d6767b902c74364eae18ed3a872"}
Oct 31 01:00:55.291046 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:55.291018    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"
Oct 31 01:00:55.393023 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:55.392984    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/734185a4-0ff7-4580-abb7-865dee21b8aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/734185a4-0ff7-4580-abb7-865dee21b8aa-tls-certs\") pod \"734185a4-0ff7-4580-abb7-865dee21b8aa\" (UID: \"734185a4-0ff7-4580-abb7-865dee21b8aa\") "
Oct 31 01:00:55.393218 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:55.393127    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-model-cache\") pod \"734185a4-0ff7-4580-abb7-865dee21b8aa\" (UID: \"734185a4-0ff7-4580-abb7-865dee21b8aa\") "
Oct 31 01:00:55.393296 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:55.393215    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-dshm\") pod \"734185a4-0ff7-4580-abb7-865dee21b8aa\" (UID: \"734185a4-0ff7-4580-abb7-865dee21b8aa\") "
Oct 31 01:00:55.393395 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:55.393377    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-tmp-dir\") pod \"734185a4-0ff7-4580-abb7-865dee21b8aa\" (UID: \"734185a4-0ff7-4580-abb7-865dee21b8aa\") "
Oct 31 01:00:55.393468 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:55.393426    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-model-cache" pod "734185a4-0ff7-4580-abb7-865dee21b8aa" (UID: "734185a4-0ff7-4580-abb7-865dee21b8aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:55.393468 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:55.393444    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/734185a4-0ff7-4580-abb7-865dee21b8aa-kube-api-access-nc7r9\" (UniqueName: \"kubernetes.io/projected/734185a4-0ff7-4580-abb7-865dee21b8aa-kube-api-access-nc7r9\") pod \"734185a4-0ff7-4580-abb7-865dee21b8aa\" (UID: \"734185a4-0ff7-4580-abb7-865dee21b8aa\") "
Oct 31 01:00:55.393576 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:55.393536    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-home\" (UniqueName: \"kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-home\") pod \"734185a4-0ff7-4580-abb7-865dee21b8aa\" (UID: \"734185a4-0ff7-4580-abb7-865dee21b8aa\") "
Oct 31 01:00:55.393716 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:55.393674    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-tmp-dir" pod "734185a4-0ff7-4580-abb7-865dee21b8aa" (UID: "734185a4-0ff7-4580-abb7-865dee21b8aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:55.393941 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:55.393905    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-home" pod "734185a4-0ff7-4580-abb7-865dee21b8aa" (UID: "734185a4-0ff7-4580-abb7-865dee21b8aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:55.394419 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:55.394393    4327 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-model-cache\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:55.394518 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:55.394426    4327 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-tmp-dir\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:55.394518 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:55.394440    4327 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-home\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:55.395787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:55.395760    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-dshm" pod "734185a4-0ff7-4580-abb7-865dee21b8aa" (UID: "734185a4-0ff7-4580-abb7-865dee21b8aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:55.395911 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:55.395889    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/734185a4-0ff7-4580-abb7-865dee21b8aa-tls-certs" pod "734185a4-0ff7-4580-abb7-865dee21b8aa" (UID: "734185a4-0ff7-4580-abb7-865dee21b8aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:00:55.396030 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:55.396003    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734185a4-0ff7-4580-abb7-865dee21b8aa-kube-api-access-nc7r9" pod "734185a4-0ff7-4580-abb7-865dee21b8aa" (UID: "734185a4-0ff7-4580-abb7-865dee21b8aa"). InnerVolumeSpecName "kube-api-access-nc7r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:00:55.496970 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:55.496895    4327 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/734185a4-0ff7-4580-abb7-865dee21b8aa-tls-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:55.496970 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:55.496936    4327 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/734185a4-0ff7-4580-abb7-865dee21b8aa-dshm\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:55.496970 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:55.496946    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc7r9\" (UniqueName: \"kubernetes.io/projected/734185a4-0ff7-4580-abb7-865dee21b8aa-kube-api-access-nc7r9\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:56.227957 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:56.227882    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"
Oct 31 01:00:56.228415 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:56.227878    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf" event={"ID":"734185a4-0ff7-4580-abb7-865dee21b8aa","Type":"ContainerDied","Data":"c2ebfc18c5d9a7285e9497c4e4b2235d12c454a218a4fc68515d3acc49ee36bb"}
Oct 31 01:00:56.228415 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:56.228003    4327 scope.go:122] "RemoveContainer" containerID="1186698632d7057827622a69c91745082dee4d6767b902c74364eae18ed3a872"
Oct 31 01:00:56.281209 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:56.281174    4327 patch_prober.go:29] interesting pod/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:00:56.281412 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:56.281235    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht" podUID="e30c1c23-92c5-47b6-a250-3519dbfae94b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:00:56.282737 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:56.282716    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"]
Oct 31 01:00:56.294595 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:56.294561    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-fd1dfc/test-model-fd1dfc-kserve-5d759978c9-74txf"]
Oct 31 01:00:57.923487 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:57.923457    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="734185a4-0ff7-4580-abb7-865dee21b8aa" path="/var/lib/kubelet/pods/734185a4-0ff7-4580-abb7-865dee21b8aa/volumes"
Oct 31 01:00:58.737934 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.737904    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 01:00:58.864247 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.864207    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e30c1c23-92c5-47b6-a250-3519dbfae94b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e30c1c23-92c5-47b6-a250-3519dbfae94b-istiod-ca-cert\") pod \"e30c1c23-92c5-47b6-a250-3519dbfae94b\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") "
Oct 31 01:00:58.864422 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.864302    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-envoy\") pod \"e30c1c23-92c5-47b6-a250-3519dbfae94b\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") "
Oct 31 01:00:58.864422 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.864386    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-workload-certs\") pod \"e30c1c23-92c5-47b6-a250-3519dbfae94b\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") "
Oct 31 01:00:58.864504 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.864449    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-data\") pod \"e30c1c23-92c5-47b6-a250-3519dbfae94b\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") "
Oct 31 01:00:58.864556 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.864539    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-workload-socket\") pod \"e30c1c23-92c5-47b6-a250-3519dbfae94b\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") "
Oct 31 01:00:58.864648 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.864633    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e30c1c23-92c5-47b6-a250-3519dbfae94b-kube-api-access-z829t\" (UniqueName: \"kubernetes.io/projected/e30c1c23-92c5-47b6-a250-3519dbfae94b-kube-api-access-z829t\") pod \"e30c1c23-92c5-47b6-a250-3519dbfae94b\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") "
Oct 31 01:00:58.864789 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.864771    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-credential-socket\") pod \"e30c1c23-92c5-47b6-a250-3519dbfae94b\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") "
Oct 31 01:00:58.864862 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.864848    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-token\" (UniqueName: \"kubernetes.io/projected/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-token\") pod \"e30c1c23-92c5-47b6-a250-3519dbfae94b\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") "
Oct 31 01:00:58.864978 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.864965    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-podinfo\") pod \"e30c1c23-92c5-47b6-a250-3519dbfae94b\" (UID: \"e30c1c23-92c5-47b6-a250-3519dbfae94b\") "
Oct 31 01:00:58.866428 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.866395    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-data" pod "e30c1c23-92c5-47b6-a250-3519dbfae94b" (UID: "e30c1c23-92c5-47b6-a250-3519dbfae94b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:58.866781 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.866755    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-credential-socket" pod "e30c1c23-92c5-47b6-a250-3519dbfae94b" (UID: "e30c1c23-92c5-47b6-a250-3519dbfae94b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:58.866927 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.866774    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-workload-socket" pod "e30c1c23-92c5-47b6-a250-3519dbfae94b" (UID: "e30c1c23-92c5-47b6-a250-3519dbfae94b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:58.867105 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.867041    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-workload-certs" pod "e30c1c23-92c5-47b6-a250-3519dbfae94b" (UID: "e30c1c23-92c5-47b6-a250-3519dbfae94b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:58.867202 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.867134    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e30c1c23-92c5-47b6-a250-3519dbfae94b-istiod-ca-cert" pod "e30c1c23-92c5-47b6-a250-3519dbfae94b" (UID: "e30c1c23-92c5-47b6-a250-3519dbfae94b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:00:58.869039 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.869008    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-envoy" pod "e30c1c23-92c5-47b6-a250-3519dbfae94b" (UID: "e30c1c23-92c5-47b6-a250-3519dbfae94b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:58.870044 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.870005    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e30c1c23-92c5-47b6-a250-3519dbfae94b-kube-api-access-z829t" pod "e30c1c23-92c5-47b6-a250-3519dbfae94b" (UID: "e30c1c23-92c5-47b6-a250-3519dbfae94b"). InnerVolumeSpecName "kube-api-access-z829t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:00:58.870198 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.870121    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-podinfo" pod "e30c1c23-92c5-47b6-a250-3519dbfae94b" (UID: "e30c1c23-92c5-47b6-a250-3519dbfae94b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:00:58.873455 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.871181    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-token" pod "e30c1c23-92c5-47b6-a250-3519dbfae94b" (UID: "e30c1c23-92c5-47b6-a250-3519dbfae94b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:00:58.979246 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.979162    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:58.979246 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.979194    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:58.979246 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.979228    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:58.979246 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.979238    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e30c1c23-92c5-47b6-a250-3519dbfae94b-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:58.979246 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.979247    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:58.979893 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.979256    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:58.979893 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.979263    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:58.979893 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.979271    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30c1c23-92c5-47b6-a250-3519dbfae94b-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:58.979893 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:58.979280    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z829t\" (UniqueName: \"kubernetes.io/projected/e30c1c23-92c5-47b6-a250-3519dbfae94b-kube-api-access-z829t\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:59.278331 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.278252    4327 generic.go:356] "Generic (PLEG): container finished" podID="e30c1c23-92c5-47b6-a250-3519dbfae94b" containerID="7a7aec5ea85a41c0f310a46441a132ed668b7b400d2679fc50f3cbd031e5cf49" exitCode=0
Oct 31 01:00:59.278538 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.278526    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"
Oct 31 01:00:59.278837 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.278795    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht" event={"ID":"e30c1c23-92c5-47b6-a250-3519dbfae94b","Type":"ContainerDied","Data":"7a7aec5ea85a41c0f310a46441a132ed668b7b400d2679fc50f3cbd031e5cf49"}
Oct 31 01:00:59.278987 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.278865    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht" event={"ID":"e30c1c23-92c5-47b6-a250-3519dbfae94b","Type":"ContainerDied","Data":"47dc15b9ee081433899a048539535a56388168370630d1534df69f4b1fb37e8d"}
Oct 31 01:00:59.278987 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.278897    4327 scope.go:122] "RemoveContainer" containerID="7a7aec5ea85a41c0f310a46441a132ed668b7b400d2679fc50f3cbd031e5cf49"
Oct 31 01:00:59.280837 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.280802    4327 generic.go:356] "Generic (PLEG): container finished" podID="a9369d90-6579-4f6a-befa-eb8a84e09bc0" containerID="96f71e10b231416b3a68ce099d40f41e8eb80842a419478e38da90a444c87566" exitCode=0
Oct 31 01:00:59.280946 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.280890    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs" event={"ID":"a9369d90-6579-4f6a-befa-eb8a84e09bc0","Type":"ContainerDied","Data":"96f71e10b231416b3a68ce099d40f41e8eb80842a419478e38da90a444c87566"}
Oct 31 01:00:59.283025 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.282999    4327 generic.go:356] "Generic (PLEG): container finished" podID="dccc262c-1c89-428b-a7a0-5d931327bf82" containerID="501f86f6c351e8129eb3ccaa001b5d72491fd4cb9ae2c59ba5f0a99233cc3406" exitCode=0
Oct 31 01:00:59.283122 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.283036    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz" event={"ID":"dccc262c-1c89-428b-a7a0-5d931327bf82","Type":"ContainerDied","Data":"501f86f6c351e8129eb3ccaa001b5d72491fd4cb9ae2c59ba5f0a99233cc3406"}
Oct 31 01:00:59.295439 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.295326    4327 scope.go:122] "RemoveContainer" containerID="ab5032d092a1c46a9a13905c1fd3dd5df965157cba66865ffc90ace375d5a265"
Oct 31 01:00:59.318347 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.318304    4327 scope.go:122] "RemoveContainer" containerID="7a7aec5ea85a41c0f310a46441a132ed668b7b400d2679fc50f3cbd031e5cf49"
Oct 31 01:00:59.319531 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:59.319189    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a7aec5ea85a41c0f310a46441a132ed668b7b400d2679fc50f3cbd031e5cf49\": container with ID starting with 7a7aec5ea85a41c0f310a46441a132ed668b7b400d2679fc50f3cbd031e5cf49 not found: ID does not exist" containerID="7a7aec5ea85a41c0f310a46441a132ed668b7b400d2679fc50f3cbd031e5cf49"
Oct 31 01:00:59.319531 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.319232    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a7aec5ea85a41c0f310a46441a132ed668b7b400d2679fc50f3cbd031e5cf49"} err="failed to get container status \"7a7aec5ea85a41c0f310a46441a132ed668b7b400d2679fc50f3cbd031e5cf49\": rpc error: code = NotFound desc = could not find container \"7a7aec5ea85a41c0f310a46441a132ed668b7b400d2679fc50f3cbd031e5cf49\": container with ID starting with 7a7aec5ea85a41c0f310a46441a132ed668b7b400d2679fc50f3cbd031e5cf49 not found: ID does not exist"
Oct 31 01:00:59.319531 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.319270    4327 scope.go:122] "RemoveContainer" containerID="ab5032d092a1c46a9a13905c1fd3dd5df965157cba66865ffc90ace375d5a265"
Oct 31 01:00:59.320172 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:00:59.319928    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab5032d092a1c46a9a13905c1fd3dd5df965157cba66865ffc90ace375d5a265\": container with ID starting with ab5032d092a1c46a9a13905c1fd3dd5df965157cba66865ffc90ace375d5a265 not found: ID does not exist" containerID="ab5032d092a1c46a9a13905c1fd3dd5df965157cba66865ffc90ace375d5a265"
Oct 31 01:00:59.320172 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.319966    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab5032d092a1c46a9a13905c1fd3dd5df965157cba66865ffc90ace375d5a265"} err="failed to get container status \"ab5032d092a1c46a9a13905c1fd3dd5df965157cba66865ffc90ace375d5a265\": rpc error: code = NotFound desc = could not find container \"ab5032d092a1c46a9a13905c1fd3dd5df965157cba66865ffc90ace375d5a265\": container with ID starting with ab5032d092a1c46a9a13905c1fd3dd5df965157cba66865ffc90ace375d5a265 not found: ID does not exist"
Oct 31 01:00:59.329760 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.329732    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs"
Oct 31 01:00:59.340105 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.340077    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"]
Oct 31 01:00:59.352149 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.352120    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-zpbht"]
Oct 31 01:00:59.385302 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.385269    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz"
Oct 31 01:00:59.485976 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.485943    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dccc262c-1c89-428b-a7a0-5d931327bf82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dccc262c-1c89-428b-a7a0-5d931327bf82-plugins-config-volume\") pod \"dccc262c-1c89-428b-a7a0-5d931327bf82\" (UID: \"dccc262c-1c89-428b-a7a0-5d931327bf82\") "
Oct 31 01:00:59.486143 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.486050    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9369d90-6579-4f6a-befa-eb8a84e09bc0-kube-api-access-4pwrt\" (UniqueName: \"kubernetes.io/projected/a9369d90-6579-4f6a-befa-eb8a84e09bc0-kube-api-access-4pwrt\") pod \"a9369d90-6579-4f6a-befa-eb8a84e09bc0\" (UID: \"a9369d90-6579-4f6a-befa-eb8a84e09bc0\") "
Oct 31 01:00:59.486262 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.486239    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dccc262c-1c89-428b-a7a0-5d931327bf82-kube-api-access-fg5sj\" (UniqueName: \"kubernetes.io/projected/dccc262c-1c89-428b-a7a0-5d931327bf82-kube-api-access-fg5sj\") pod \"dccc262c-1c89-428b-a7a0-5d931327bf82\" (UID: \"dccc262c-1c89-428b-a7a0-5d931327bf82\") "
Oct 31 01:00:59.486377 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.486357    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9369d90-6579-4f6a-befa-eb8a84e09bc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9369d90-6579-4f6a-befa-eb8a84e09bc0-plugins-config-volume\") pod \"a9369d90-6579-4f6a-befa-eb8a84e09bc0\" (UID: \"a9369d90-6579-4f6a-befa-eb8a84e09bc0\") "
Oct 31 01:00:59.486377 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.486373    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dccc262c-1c89-428b-a7a0-5d931327bf82-plugins-config-volume" pod "dccc262c-1c89-428b-a7a0-5d931327bf82" (UID: "dccc262c-1c89-428b-a7a0-5d931327bf82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:00:59.486758 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.486728    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9369d90-6579-4f6a-befa-eb8a84e09bc0-plugins-config-volume" pod "a9369d90-6579-4f6a-befa-eb8a84e09bc0" (UID: "a9369d90-6579-4f6a-befa-eb8a84e09bc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:00:59.487098 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.487070    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dccc262c-1c89-428b-a7a0-5d931327bf82-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:59.487098 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.487099    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9369d90-6579-4f6a-befa-eb8a84e09bc0-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:59.488541 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.488511    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9369d90-6579-4f6a-befa-eb8a84e09bc0-kube-api-access-4pwrt" pod "a9369d90-6579-4f6a-befa-eb8a84e09bc0" (UID: "a9369d90-6579-4f6a-befa-eb8a84e09bc0"). InnerVolumeSpecName "kube-api-access-4pwrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:00:59.488632 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.488574    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dccc262c-1c89-428b-a7a0-5d931327bf82-kube-api-access-fg5sj" pod "dccc262c-1c89-428b-a7a0-5d931327bf82" (UID: "dccc262c-1c89-428b-a7a0-5d931327bf82"). InnerVolumeSpecName "kube-api-access-fg5sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:00:59.589103 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.589006    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pwrt\" (UniqueName: \"kubernetes.io/projected/a9369d90-6579-4f6a-befa-eb8a84e09bc0-kube-api-access-4pwrt\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:59.589103 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.589040    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg5sj\" (UniqueName: \"kubernetes.io/projected/dccc262c-1c89-428b-a7a0-5d931327bf82-kube-api-access-fg5sj\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:00:59.925033 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:00:59.925003    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e30c1c23-92c5-47b6-a250-3519dbfae94b" path="/var/lib/kubelet/pods/e30c1c23-92c5-47b6-a250-3519dbfae94b/volumes"
Oct 31 01:01:00.292569 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:00.292488    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs"
Oct 31 01:01:00.292569 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:00.292537    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs" event={"ID":"a9369d90-6579-4f6a-befa-eb8a84e09bc0","Type":"ContainerDied","Data":"83a0fe4c92a8d1f68033330cbe0a8708c565e51603e16d153b0bb8b83961babf"}
Oct 31 01:01:00.293257 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:00.292625    4327 scope.go:122] "RemoveContainer" containerID="96f71e10b231416b3a68ce099d40f41e8eb80842a419478e38da90a444c87566"
Oct 31 01:01:00.294450 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:00.294426    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz"
Oct 31 01:01:00.294568 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:00.294483    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz" event={"ID":"dccc262c-1c89-428b-a7a0-5d931327bf82","Type":"ContainerDied","Data":"5b97e85a12ea4646d32dd787bcad0134025f074ec0ac20abaae10394f3fd4976"}
Oct 31 01:01:00.304419 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:00.304399    4327 scope.go:122] "RemoveContainer" containerID="501f86f6c351e8129eb3ccaa001b5d72491fd4cb9ae2c59ba5f0a99233cc3406"
Oct 31 01:01:00.348775 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:00.348743    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz"]
Oct 31 01:01:00.353416 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:00.353396    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d5621-5c496fd68c-mmnpz"]
Oct 31 01:01:00.411026 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:00.410985    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs"]
Oct 31 01:01:00.418408 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:00.418370    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d5621-6747cc9b4f-7ncgs"]
Oct 31 01:01:01.925141 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:01.925106    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9369d90-6579-4f6a-befa-eb8a84e09bc0" path="/var/lib/kubelet/pods/a9369d90-6579-4f6a-befa-eb8a84e09bc0/volumes"
Oct 31 01:01:01.925618 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:01.925603    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dccc262c-1c89-428b-a7a0-5d931327bf82" path="/var/lib/kubelet/pods/dccc262c-1c89-428b-a7a0-5d931327bf82/volumes"
Oct 31 01:01:04.611990 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:04.611753    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd"]
Oct 31 01:01:04.649721 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:04.649653    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9"]
Oct 31 01:01:09.613355 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:09.613311    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd" podUID="d628dce5-6346-46cc-a356-ad526ffd0b96" containerName="payload-processing" containerID="cri-o://d77a8c61c24b16dbc4269489c7d97b64eacdff26c7c1922c2043e0633389359c" gracePeriod=25
Oct 31 01:01:09.650663 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:09.650600    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9" podUID="67570d88-4047-46e3-9c60-3522e8c59493" containerName="payload-pre-processing" containerID="cri-o://a54e5003fd0d96d98aca740d43e27a7c303bd014adf735ec1b9e5370510b72bc" gracePeriod=25
Oct 31 01:01:11.778295 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:11.778243    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc"]
Oct 31 01:01:11.802918 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:11.802880    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28"]
Oct 31 01:01:15.300975 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:01:15.300930    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:01:16.370800 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:16.351405    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"]
Oct 31 01:01:16.371895 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:16.371857    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"]
Oct 31 01:01:16.372317 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:16.372287    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd" podUID="6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3" containerName="istio-proxy" containerID="cri-o://b3268d824f6459a5ce45f602b372e77676b9abc1eee0ed4dc6232c980ca0d3bf" gracePeriod=30
Oct 31 01:01:16.377587 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:16.372851    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz" podUID="820c95b2-cfd6-4893-80ed-1c746d4cfef2" containerName="istio-proxy" containerID="cri-o://a1711b8c06eb31b128c553f8ccd5bd0a61636ff87d8c23bfe20688aaedc373a4" gracePeriod=30
Oct 31 01:01:16.397578 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:16.397535    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"]
Oct 31 01:01:16.397854 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:16.397823    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf" podUID="a5f7f289-ef35-494d-91c9-034c7f1294d4" containerName="istio-proxy" containerID="cri-o://d7c679a4f978fe103be5bb8f73307dc8a715d9159088dcc7e6c4c82e148cfa71" gracePeriod=30
Oct 31 01:01:16.783916 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:16.783796    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc" podUID="841015e0-30ae-4810-a913-598be430b8fd" containerName="payload-processing" containerID="cri-o://a1f7c1812ca2d57653d4c723de70bff882d8f7d43871abd93487ddd144887b20" gracePeriod=25
Oct 31 01:01:16.803188 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:16.803137    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28" podUID="f113a57a-325b-4508-ad0f-3e9677b3f95d" containerName="payload-pre-processing" containerID="cri-o://d2b98a8a41303c17b12e9bdfb78a79c9bb640cbf7d68c8354e1dd9c32104d5b1" gracePeriod=25
Oct 31 01:01:19.527034 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:19.526992    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9" podUID="67570d88-4047-46e3-9c60-3522e8c59493" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:01:19.528472 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:19.528441    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd" podUID="d628dce5-6346-46cc-a356-ad526ffd0b96" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:01:20.228104 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.228057    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc" podUID="841015e0-30ae-4810-a913-598be430b8fd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:01:20.312697 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.312648    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28" podUID="f113a57a-325b-4508-ad0f-3e9677b3f95d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:01:20.362758 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.362721    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"]
Oct 31 01:01:20.363628 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363612    4327 cpu_manager.go:414] "containerMap: removing container" podUID="d352532d-5b9b-4713-a13c-32cd6d2c7716" containerName="cleanup"
Oct 31 01:01:20.363702 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363629    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d352532d-5b9b-4713-a13c-32cd6d2c7716" containerName="cleanup"
Oct 31 01:01:20.363702 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363643    4327 cpu_manager.go:414] "containerMap: removing container" podUID="11188116-a59b-48f1-9f7d-85d09032455e" containerName="cleanup"
Oct 31 01:01:20.363702 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363648    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11188116-a59b-48f1-9f7d-85d09032455e" containerName="cleanup"
Oct 31 01:01:20.363702 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363664    4327 cpu_manager.go:414] "containerMap: removing container" podUID="7f085592-c63f-433e-a77f-0b0c20746fe2" containerName="cleanup"
Oct 31 01:01:20.363702 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363672    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f085592-c63f-433e-a77f-0b0c20746fe2" containerName="cleanup"
Oct 31 01:01:20.363702 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363691    4327 cpu_manager.go:414] "containerMap: removing container" podUID="8b6d9a4f-7b06-42f1-8e32-720489b7ba93" containerName="cleanup"
Oct 31 01:01:20.363702 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363696    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b6d9a4f-7b06-42f1-8e32-720489b7ba93" containerName="cleanup"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363709    4327 cpu_manager.go:414] "containerMap: removing container" podUID="13752887-b28f-49f8-855d-2eed4657dc61" containerName="cleanup"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363715    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13752887-b28f-49f8-855d-2eed4657dc61" containerName="cleanup"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363724    4327 cpu_manager.go:414] "containerMap: removing container" podUID="11188116-a59b-48f1-9f7d-85d09032455e" containerName="cleanup"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363728    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11188116-a59b-48f1-9f7d-85d09032455e" containerName="cleanup"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363736    4327 cpu_manager.go:414] "containerMap: removing container" podUID="e30c1c23-92c5-47b6-a250-3519dbfae94b" containerName="istio-proxy"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363740    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e30c1c23-92c5-47b6-a250-3519dbfae94b" containerName="istio-proxy"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363748    4327 cpu_manager.go:414] "containerMap: removing container" podUID="c4ee6cb0-2737-426b-9f00-4d9f550c8164" containerName="cleanup"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363751    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ee6cb0-2737-426b-9f00-4d9f550c8164" containerName="cleanup"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363757    4327 cpu_manager.go:414] "containerMap: removing container" podUID="7f085592-c63f-433e-a77f-0b0c20746fe2" containerName="cleanup"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363761    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f085592-c63f-433e-a77f-0b0c20746fe2" containerName="cleanup"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363768    4327 cpu_manager.go:414] "containerMap: removing container" podUID="13752887-b28f-49f8-855d-2eed4657dc61" containerName="cleanup"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363772    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13752887-b28f-49f8-855d-2eed4657dc61" containerName="cleanup"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363784    4327 cpu_manager.go:414] "containerMap: removing container" podUID="b118e1e7-3dd9-4107-b740-9032c60dee40" containerName="cleanup"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363789    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b118e1e7-3dd9-4107-b740-9032c60dee40" containerName="cleanup"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363803    4327 cpu_manager.go:414] "containerMap: removing container" podUID="f45f589e-69d1-49fd-b27d-a15d1f886235" containerName="cleanup"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363807    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45f589e-69d1-49fd-b27d-a15d1f886235" containerName="cleanup"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363812    4327 cpu_manager.go:414] "containerMap: removing container" podUID="f45f589e-69d1-49fd-b27d-a15d1f886235" containerName="cleanup"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363816    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45f589e-69d1-49fd-b27d-a15d1f886235" containerName="cleanup"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363823    4327 cpu_manager.go:414] "containerMap: removing container" podUID="8b6d9a4f-7b06-42f1-8e32-720489b7ba93" containerName="cleanup"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363827    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b6d9a4f-7b06-42f1-8e32-720489b7ba93" containerName="cleanup"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363842    4327 cpu_manager.go:414] "containerMap: removing container" podUID="b118e1e7-3dd9-4107-b740-9032c60dee40" containerName="cleanup"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363847    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b118e1e7-3dd9-4107-b740-9032c60dee40" containerName="cleanup"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363853    4327 cpu_manager.go:414] "containerMap: removing container" podUID="dccc262c-1c89-428b-a7a0-5d931327bf82" containerName="payload-pre-processing"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363865    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dccc262c-1c89-428b-a7a0-5d931327bf82" containerName="payload-pre-processing"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363872    4327 cpu_manager.go:414] "containerMap: removing container" podUID="a9369d90-6579-4f6a-befa-eb8a84e09bc0" containerName="payload-processing"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363878    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9369d90-6579-4f6a-befa-eb8a84e09bc0" containerName="payload-processing"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363886    4327 cpu_manager.go:414] "containerMap: removing container" podUID="a175adcd-0c7a-4572-940c-96eca0ee58d5" containerName="main"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363890    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a175adcd-0c7a-4572-940c-96eca0ee58d5" containerName="main"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363898    4327 cpu_manager.go:414] "containerMap: removing container" podUID="734185a4-0ff7-4580-abb7-865dee21b8aa" containerName="main"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363903    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="734185a4-0ff7-4580-abb7-865dee21b8aa" containerName="main"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363914    4327 cpu_manager.go:414] "containerMap: removing container" podUID="d352532d-5b9b-4713-a13c-32cd6d2c7716" containerName="cleanup"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363917    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d352532d-5b9b-4713-a13c-32cd6d2c7716" containerName="cleanup"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363933    4327 cpu_manager.go:414] "containerMap: removing container" podUID="e30c1c23-92c5-47b6-a250-3519dbfae94b" containerName="istio-proxy"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363937    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e30c1c23-92c5-47b6-a250-3519dbfae94b" containerName="istio-proxy"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363945    4327 cpu_manager.go:414] "containerMap: removing container" podUID="c4ee6cb0-2737-426b-9f00-4d9f550c8164" containerName="cleanup"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363950    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ee6cb0-2737-426b-9f00-4d9f550c8164" containerName="cleanup"
Oct 31 01:01:20.363939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363957    4327 cpu_manager.go:414] "containerMap: removing container" podUID="2a45ba96-dddd-43f5-9e50-0b08194799f8" containerName="istio-proxy"
Oct 31 01:01:20.365038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.363961    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a45ba96-dddd-43f5-9e50-0b08194799f8" containerName="istio-proxy"
Oct 31 01:01:20.365038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.364078    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="d352532d-5b9b-4713-a13c-32cd6d2c7716" containerName="cleanup"
Oct 31 01:01:20.365038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.364086    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="11188116-a59b-48f1-9f7d-85d09032455e" containerName="cleanup"
Oct 31 01:01:20.365038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.364096    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f085592-c63f-433e-a77f-0b0c20746fe2" containerName="cleanup"
Oct 31 01:01:20.365038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.364105    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b6d9a4f-7b06-42f1-8e32-720489b7ba93" containerName="cleanup"
Oct 31 01:01:20.365038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.364114    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="13752887-b28f-49f8-855d-2eed4657dc61" containerName="cleanup"
Oct 31 01:01:20.365038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.364123    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f085592-c63f-433e-a77f-0b0c20746fe2" containerName="cleanup"
Oct 31 01:01:20.365038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.364133    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="11188116-a59b-48f1-9f7d-85d09032455e" containerName="cleanup"
Oct 31 01:01:20.365038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.364140    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="e30c1c23-92c5-47b6-a250-3519dbfae94b" containerName="istio-proxy"
Oct 31 01:01:20.365038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.364146    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4ee6cb0-2737-426b-9f00-4d9f550c8164" containerName="cleanup"
Oct 31 01:01:20.365038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.364151    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="13752887-b28f-49f8-855d-2eed4657dc61" containerName="cleanup"
Oct 31 01:01:20.365038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.364157    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="b118e1e7-3dd9-4107-b740-9032c60dee40" containerName="cleanup"
Oct 31 01:01:20.365038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.364164    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45f589e-69d1-49fd-b27d-a15d1f886235" containerName="cleanup"
Oct 31 01:01:20.365038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.364168    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45f589e-69d1-49fd-b27d-a15d1f886235" containerName="cleanup"
Oct 31 01:01:20.365038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.364174    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b6d9a4f-7b06-42f1-8e32-720489b7ba93" containerName="cleanup"
Oct 31 01:01:20.365038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.364180    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4ee6cb0-2737-426b-9f00-4d9f550c8164" containerName="cleanup"
Oct 31 01:01:20.365038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.364185    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="b118e1e7-3dd9-4107-b740-9032c60dee40" containerName="cleanup"
Oct 31 01:01:20.365038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.364190    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="dccc262c-1c89-428b-a7a0-5d931327bf82" containerName="payload-pre-processing"
Oct 31 01:01:20.365038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.364197    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9369d90-6579-4f6a-befa-eb8a84e09bc0" containerName="payload-processing"
Oct 31 01:01:20.365038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.364203    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="a175adcd-0c7a-4572-940c-96eca0ee58d5" containerName="main"
Oct 31 01:01:20.365038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.364210    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="734185a4-0ff7-4580-abb7-865dee21b8aa" containerName="main"
Oct 31 01:01:20.365038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.364216    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b6d9a4f-7b06-42f1-8e32-720489b7ba93" containerName="cleanup"
Oct 31 01:01:20.365038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.364222    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="d352532d-5b9b-4713-a13c-32cd6d2c7716" containerName="cleanup"
Oct 31 01:01:20.365038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.364228    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4ee6cb0-2737-426b-9f00-4d9f550c8164" containerName="cleanup"
Oct 31 01:01:20.365038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.364235    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a45ba96-dddd-43f5-9e50-0b08194799f8" containerName="istio-proxy"
Oct 31 01:01:20.370763 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.370732    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"
Oct 31 01:01:20.393556 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.393447    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"]
Oct 31 01:01:20.397987 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.397951    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6f8856\"/\"test-model-6f8856-kserve-self-signed-certs\""
Oct 31 01:01:20.400546 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.400521    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6f8856\"/\"openshift-service-ca.crt\""
Oct 31 01:01:20.400546 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.400537    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6f8856\"/\"kube-root-ca.crt\""
Oct 31 01:01:20.401002 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.400547    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6f8856\"/\"default-dockercfg-hxp2t\""
Oct 31 01:01:20.433696 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.433645    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-model-cache\") pod \"test-model-6f8856-kserve-f767f9b45-7qmlc\" (UID: \"9abe1d77-158a-4806-85f5-954fc5854b06\") " pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"
Oct 31 01:01:20.433890 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.433756    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2s4z\" (UniqueName: \"kubernetes.io/projected/9abe1d77-158a-4806-85f5-954fc5854b06-kube-api-access-r2s4z\") pod \"test-model-6f8856-kserve-f767f9b45-7qmlc\" (UID: \"9abe1d77-158a-4806-85f5-954fc5854b06\") " pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"
Oct 31 01:01:20.433890 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.433865    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-tmp-dir\") pod \"test-model-6f8856-kserve-f767f9b45-7qmlc\" (UID: \"9abe1d77-158a-4806-85f5-954fc5854b06\") " pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"
Oct 31 01:01:20.433890 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.433885    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9abe1d77-158a-4806-85f5-954fc5854b06-tls-certs\") pod \"test-model-6f8856-kserve-f767f9b45-7qmlc\" (UID: \"9abe1d77-158a-4806-85f5-954fc5854b06\") " pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"
Oct 31 01:01:20.434067 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.433969    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-dshm\") pod \"test-model-6f8856-kserve-f767f9b45-7qmlc\" (UID: \"9abe1d77-158a-4806-85f5-954fc5854b06\") " pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"
Oct 31 01:01:20.434067 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.433992    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-home\") pod \"test-model-6f8856-kserve-f767f9b45-7qmlc\" (UID: \"9abe1d77-158a-4806-85f5-954fc5854b06\") " pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"
Oct 31 01:01:20.535152 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.535065    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-dshm\") pod \"test-model-6f8856-kserve-f767f9b45-7qmlc\" (UID: \"9abe1d77-158a-4806-85f5-954fc5854b06\") " pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"
Oct 31 01:01:20.535152 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.535109    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-home\") pod \"test-model-6f8856-kserve-f767f9b45-7qmlc\" (UID: \"9abe1d77-158a-4806-85f5-954fc5854b06\") " pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"
Oct 31 01:01:20.535741 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.535185    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-model-cache\") pod \"test-model-6f8856-kserve-f767f9b45-7qmlc\" (UID: \"9abe1d77-158a-4806-85f5-954fc5854b06\") " pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"
Oct 31 01:01:20.535741 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.535222    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2s4z\" (UniqueName: \"kubernetes.io/projected/9abe1d77-158a-4806-85f5-954fc5854b06-kube-api-access-r2s4z\") pod \"test-model-6f8856-kserve-f767f9b45-7qmlc\" (UID: \"9abe1d77-158a-4806-85f5-954fc5854b06\") " pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"
Oct 31 01:01:20.535741 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.535341    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-tmp-dir\") pod \"test-model-6f8856-kserve-f767f9b45-7qmlc\" (UID: \"9abe1d77-158a-4806-85f5-954fc5854b06\") " pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"
Oct 31 01:01:20.535741 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.535366    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9abe1d77-158a-4806-85f5-954fc5854b06-tls-certs\") pod \"test-model-6f8856-kserve-f767f9b45-7qmlc\" (UID: \"9abe1d77-158a-4806-85f5-954fc5854b06\") " pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"
Oct 31 01:01:20.535741 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.535591    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-home\") pod \"test-model-6f8856-kserve-f767f9b45-7qmlc\" (UID: \"9abe1d77-158a-4806-85f5-954fc5854b06\") " pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"
Oct 31 01:01:20.535741 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.535631    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-model-cache\") pod \"test-model-6f8856-kserve-f767f9b45-7qmlc\" (UID: \"9abe1d77-158a-4806-85f5-954fc5854b06\") " pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"
Oct 31 01:01:20.535977 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.535745    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-tmp-dir\") pod \"test-model-6f8856-kserve-f767f9b45-7qmlc\" (UID: \"9abe1d77-158a-4806-85f5-954fc5854b06\") " pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"
Oct 31 01:01:20.537400 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.537382    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-dshm\") pod \"test-model-6f8856-kserve-f767f9b45-7qmlc\" (UID: \"9abe1d77-158a-4806-85f5-954fc5854b06\") " pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"
Oct 31 01:01:20.537855 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.537837    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9abe1d77-158a-4806-85f5-954fc5854b06-tls-certs\") pod \"test-model-6f8856-kserve-f767f9b45-7qmlc\" (UID: \"9abe1d77-158a-4806-85f5-954fc5854b06\") " pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"
Oct 31 01:01:20.559810 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.559782    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2s4z\" (UniqueName: \"kubernetes.io/projected/9abe1d77-158a-4806-85f5-954fc5854b06-kube-api-access-r2s4z\") pod \"test-model-6f8856-kserve-f767f9b45-7qmlc\" (UID: \"9abe1d77-158a-4806-85f5-954fc5854b06\") " pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"
Oct 31 01:01:20.707037 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.706998    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"
Oct 31 01:01:20.944524 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.944489    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4qhrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:01:20.944752 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.944547    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd" podUID="6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:01:20.992155 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.992115    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hk9qf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:01:20.992298 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:20.992199    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf" podUID="a5f7f289-ef35-494d-91c9-034c7f1294d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:01:21.053822 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:01:21.053794    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9abe1d77_158a_4806_85f5_954fc5854b06.slice/crio-058624440cec8b6f586f1687d39e948f047de543081a45f25c274f0daf86c51c WatchSource:0}: Error finding container 058624440cec8b6f586f1687d39e948f047de543081a45f25c274f0daf86c51c: Status 404 returned error can't find the container with id 058624440cec8b6f586f1687d39e948f047de543081a45f25c274f0daf86c51c
Oct 31 01:01:21.055347 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.055322    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"]
Oct 31 01:01:21.604562 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.604483    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc" event={"ID":"9abe1d77-158a-4806-85f5-954fc5854b06","Type":"ContainerStarted","Data":"dfd897c8e48cefc309faf4fbe68d487be9f39f64026cf60ae7296fb3c6d3f141"}
Oct 31 01:01:21.604562 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.604547    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc" event={"ID":"9abe1d77-158a-4806-85f5-954fc5854b06","Type":"ContainerStarted","Data":"058624440cec8b6f586f1687d39e948f047de543081a45f25c274f0daf86c51c"}
Oct 31 01:01:21.608670 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.608463    4327 generic.go:356] "Generic (PLEG): container finished" podID="820c95b2-cfd6-4893-80ed-1c746d4cfef2" containerID="a1711b8c06eb31b128c553f8ccd5bd0a61636ff87d8c23bfe20688aaedc373a4" exitCode=0
Oct 31 01:01:21.608670 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.608515    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz" event={"ID":"820c95b2-cfd6-4893-80ed-1c746d4cfef2","Type":"ContainerDied","Data":"a1711b8c06eb31b128c553f8ccd5bd0a61636ff87d8c23bfe20688aaedc373a4"}
Oct 31 01:01:21.613416 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.613386    4327 generic.go:356] "Generic (PLEG): container finished" podID="6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3" containerID="b3268d824f6459a5ce45f602b372e77676b9abc1eee0ed4dc6232c980ca0d3bf" exitCode=0
Oct 31 01:01:21.613612 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.613469    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd" event={"ID":"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3","Type":"ContainerDied","Data":"b3268d824f6459a5ce45f602b372e77676b9abc1eee0ed4dc6232c980ca0d3bf"}
Oct 31 01:01:21.615435 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.615406    4327 generic.go:356] "Generic (PLEG): container finished" podID="a5f7f289-ef35-494d-91c9-034c7f1294d4" containerID="d7c679a4f978fe103be5bb8f73307dc8a715d9159088dcc7e6c4c82e148cfa71" exitCode=0
Oct 31 01:01:21.615542 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.615470    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf" event={"ID":"a5f7f289-ef35-494d-91c9-034c7f1294d4","Type":"ContainerDied","Data":"d7c679a4f978fe103be5bb8f73307dc8a715d9159088dcc7e6c4c82e148cfa71"}
Oct 31 01:01:21.653023 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.652906    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc" podStartSLOduration=1.652891829 podStartE2EDuration="1.652891829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:01:21.647732753 +0000 UTC m=+2541.869768296" watchObservedRunningTime="2026-10-31 01:01:21.652891829 +0000 UTC m=+2541.874927370"
Oct 31 01:01:21.708485 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.708418    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"
Oct 31 01:01:21.729046 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.729014    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 01:01:21.835304 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.835259    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-token\" (UniqueName: \"kubernetes.io/projected/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-token\") pod \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") "
Oct 31 01:01:21.835304 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.835316    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istiod-ca-cert\") pod \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") "
Oct 31 01:01:21.835565 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.835387    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-credential-socket\") pod \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") "
Oct 31 01:01:21.835565 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.835499    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-workload-certs\") pod \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") "
Oct 31 01:01:21.835565 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.835531    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-workload-socket\") pod \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") "
Oct 31 01:01:21.835797 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.835623    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-podinfo\") pod \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") "
Oct 31 01:01:21.835797 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.835657    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-data\") pod \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") "
Oct 31 01:01:21.835917 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.835795    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-envoy\") pod \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") "
Oct 31 01:01:21.835979 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.835950    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/820c95b2-cfd6-4893-80ed-1c746d4cfef2-kube-api-access-nfdfk\" (UniqueName: \"kubernetes.io/projected/820c95b2-cfd6-4893-80ed-1c746d4cfef2-kube-api-access-nfdfk\") pod \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\" (UID: \"820c95b2-cfd6-4893-80ed-1c746d4cfef2\") "
Oct 31 01:01:21.838004 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.837968    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-workload-socket" pod "820c95b2-cfd6-4893-80ed-1c746d4cfef2" (UID: "820c95b2-cfd6-4893-80ed-1c746d4cfef2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:21.838693 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.838660    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-workload-certs" pod "820c95b2-cfd6-4893-80ed-1c746d4cfef2" (UID: "820c95b2-cfd6-4893-80ed-1c746d4cfef2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:21.838924 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.838859    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-data" pod "820c95b2-cfd6-4893-80ed-1c746d4cfef2" (UID: "820c95b2-cfd6-4893-80ed-1c746d4cfef2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:21.839043 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.839018    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istiod-ca-cert" pod "820c95b2-cfd6-4893-80ed-1c746d4cfef2" (UID: "820c95b2-cfd6-4893-80ed-1c746d4cfef2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:01:21.839268 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.839234    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-credential-socket" pod "820c95b2-cfd6-4893-80ed-1c746d4cfef2" (UID: "820c95b2-cfd6-4893-80ed-1c746d4cfef2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:21.840866 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.840832    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/820c95b2-cfd6-4893-80ed-1c746d4cfef2-kube-api-access-nfdfk" pod "820c95b2-cfd6-4893-80ed-1c746d4cfef2" (UID: "820c95b2-cfd6-4893-80ed-1c746d4cfef2"). InnerVolumeSpecName "kube-api-access-nfdfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:01:21.841252 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.841232    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-token" pod "820c95b2-cfd6-4893-80ed-1c746d4cfef2" (UID: "820c95b2-cfd6-4893-80ed-1c746d4cfef2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:01:21.842496 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.842467    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-envoy" pod "820c95b2-cfd6-4893-80ed-1c746d4cfef2" (UID: "820c95b2-cfd6-4893-80ed-1c746d4cfef2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:21.842673 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.842654    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-podinfo" pod "820c95b2-cfd6-4893-80ed-1c746d4cfef2" (UID: "820c95b2-cfd6-4893-80ed-1c746d4cfef2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:01:21.885073 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.885036    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 01:01:21.903005 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.902958    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 01:01:21.938516 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.938486    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfdfk\" (UniqueName: \"kubernetes.io/projected/820c95b2-cfd6-4893-80ed-1c746d4cfef2-kube-api-access-nfdfk\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:21.938516 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.938514    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:21.938666 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.938527    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:21.938666 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.938535    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:21.938666 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.938546    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:21.938666 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.938554    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:21.938666 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.938562    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:21.938666 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.938570    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:21.938666 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:21.938578    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/820c95b2-cfd6-4893-80ed-1c746d4cfef2-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:22.039687 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.039655    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-token\" (UniqueName: \"kubernetes.io/projected/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-token\") pod \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") "
Oct 31 01:01:22.039887 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.039741    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istiod-ca-cert\") pod \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") "
Oct 31 01:01:22.039887 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.039792    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-envoy\") pod \"a5f7f289-ef35-494d-91c9-034c7f1294d4\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") "
Oct 31 01:01:22.039887 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.039833    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-workload-certs\") pod \"a5f7f289-ef35-494d-91c9-034c7f1294d4\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") "
Oct 31 01:01:22.040086 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.040007    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f7f289-ef35-494d-91c9-034c7f1294d4-kube-api-access-f877v\" (UniqueName: \"kubernetes.io/projected/a5f7f289-ef35-494d-91c9-034c7f1294d4-kube-api-access-f877v\") pod \"a5f7f289-ef35-494d-91c9-034c7f1294d4\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") "
Oct 31 01:01:22.040086 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.040043    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-token\" (UniqueName: \"kubernetes.io/projected/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-token\") pod \"a5f7f289-ef35-494d-91c9-034c7f1294d4\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") "
Oct 31 01:01:22.040202 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.040099    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-workload-socket\") pod \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") "
Oct 31 01:01:22.040202 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.040131    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-kube-api-access-s52gp\" (UniqueName: \"kubernetes.io/projected/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-kube-api-access-s52gp\") pod \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") "
Oct 31 01:01:22.040202 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.040187    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-workload-socket\") pod \"a5f7f289-ef35-494d-91c9-034c7f1294d4\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") "
Oct 31 01:01:22.040367 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.040208    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-workload-certs" pod "a5f7f289-ef35-494d-91c9-034c7f1294d4" (UID: "a5f7f289-ef35-494d-91c9-034c7f1294d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:22.040367 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.040227    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istiod-ca-cert" pod "6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3" (UID: "6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:01:22.040367 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.040249    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-podinfo\") pod \"a5f7f289-ef35-494d-91c9-034c7f1294d4\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") "
Oct 31 01:01:22.040367 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.040301    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-credential-socket\") pod \"a5f7f289-ef35-494d-91c9-034c7f1294d4\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") "
Oct 31 01:01:22.040367 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.040337    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-workload-certs\") pod \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") "
Oct 31 01:01:22.040367 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.040362    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-podinfo\") pod \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") "
Oct 31 01:01:22.040691 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.040422    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-data\") pod \"a5f7f289-ef35-494d-91c9-034c7f1294d4\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") "
Oct 31 01:01:22.040691 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.040493    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-workload-socket" pod "6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3" (UID: "6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:22.040691 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.040509    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-data\") pod \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") "
Oct 31 01:01:22.040691 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.040550    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-envoy\") pod \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") "
Oct 31 01:01:22.040691 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.040599    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-credential-socket\") pod \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\" (UID: \"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3\") "
Oct 31 01:01:22.040965 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.040707    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f7f289-ef35-494d-91c9-034c7f1294d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f7f289-ef35-494d-91c9-034c7f1294d4-istiod-ca-cert\") pod \"a5f7f289-ef35-494d-91c9-034c7f1294d4\" (UID: \"a5f7f289-ef35-494d-91c9-034c7f1294d4\") "
Oct 31 01:01:22.040965 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.040722    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-credential-socket" pod "a5f7f289-ef35-494d-91c9-034c7f1294d4" (UID: "a5f7f289-ef35-494d-91c9-034c7f1294d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:22.041338 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.041308    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-workload-socket" pod "a5f7f289-ef35-494d-91c9-034c7f1294d4" (UID: "a5f7f289-ef35-494d-91c9-034c7f1294d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:22.041857 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.041830    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:22.041952 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.041860    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:22.041952 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.041878    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:22.041952 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.041891    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:22.041952 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.041905    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:22.042569 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.042544    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-credential-socket" pod "6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3" (UID: "6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:22.042673 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.042622    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f7f289-ef35-494d-91c9-034c7f1294d4-istiod-ca-cert" pod "a5f7f289-ef35-494d-91c9-034c7f1294d4" (UID: "a5f7f289-ef35-494d-91c9-034c7f1294d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:01:22.042788 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.042763    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-workload-certs" pod "6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3" (UID: "6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:22.043385 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.043359    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-data" pod "6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3" (UID: "6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:22.043492 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.043457    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-envoy" pod "a5f7f289-ef35-494d-91c9-034c7f1294d4" (UID: "a5f7f289-ef35-494d-91c9-034c7f1294d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:22.043555 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.043492    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-data" pod "a5f7f289-ef35-494d-91c9-034c7f1294d4" (UID: "a5f7f289-ef35-494d-91c9-034c7f1294d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:22.044044 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.044020    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-kube-api-access-s52gp" pod "6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3" (UID: "6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3"). InnerVolumeSpecName "kube-api-access-s52gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:01:22.044266 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.044245    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-token" pod "6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3" (UID: "6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:01:22.044344 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.044261    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-podinfo" pod "a5f7f289-ef35-494d-91c9-034c7f1294d4" (UID: "a5f7f289-ef35-494d-91c9-034c7f1294d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:01:22.044537 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.044514    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f7f289-ef35-494d-91c9-034c7f1294d4-kube-api-access-f877v" pod "a5f7f289-ef35-494d-91c9-034c7f1294d4" (UID: "a5f7f289-ef35-494d-91c9-034c7f1294d4"). InnerVolumeSpecName "kube-api-access-f877v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:01:22.045153 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.045133    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-token" pod "a5f7f289-ef35-494d-91c9-034c7f1294d4" (UID: "a5f7f289-ef35-494d-91c9-034c7f1294d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:01:22.045375 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.045353    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-podinfo" pod "6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3" (UID: "6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:01:22.045615 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.045589    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-envoy" pod "6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3" (UID: "6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:22.143275 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.143236    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s52gp\" (UniqueName: \"kubernetes.io/projected/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-kube-api-access-s52gp\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:22.143275 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.143269    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:22.143275 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.143279    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:22.143520 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.143288    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:22.143520 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.143297    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:22.143520 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.143306    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:22.143520 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.143313    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:22.143520 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.143321    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:22.143520 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.143329    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f7f289-ef35-494d-91c9-034c7f1294d4-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:22.143520 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.143337    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:22.143520 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.143344    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:22.143520 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.143352    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f877v\" (UniqueName: \"kubernetes.io/projected/a5f7f289-ef35-494d-91c9-034c7f1294d4-kube-api-access-f877v\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:22.143520 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.143359    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f7f289-ef35-494d-91c9-034c7f1294d4-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:22.626198 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.626167    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"
Oct 31 01:01:22.626198 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.626176    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf" event={"ID":"a5f7f289-ef35-494d-91c9-034c7f1294d4","Type":"ContainerDied","Data":"0382969f169860e646ea6ce5ab71e97658d7d4a8cf3d2f6668684e9c6adf6bc8"}
Oct 31 01:01:22.626746 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.626246    4327 scope.go:122] "RemoveContainer" containerID="d7c679a4f978fe103be5bb8f73307dc8a715d9159088dcc7e6c4c82e148cfa71"
Oct 31 01:01:22.628086 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.628066    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"
Oct 31 01:01:22.628211 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.628101    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz" event={"ID":"820c95b2-cfd6-4893-80ed-1c746d4cfef2","Type":"ContainerDied","Data":"092482c52236f71627a23b8199cb99e70a37456af0a6796222364e9f44f23d99"}
Oct 31 01:01:22.629885 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.629861    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"
Oct 31 01:01:22.630023 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.629893    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd" event={"ID":"6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3","Type":"ContainerDied","Data":"204b13b58dd9dd1620f2cc5497bd95645eb292ed57d9a517fd628440ee5510ce"}
Oct 31 01:01:22.644391 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.644368    4327 scope.go:122] "RemoveContainer" containerID="a1711b8c06eb31b128c553f8ccd5bd0a61636ff87d8c23bfe20688aaedc373a4"
Oct 31 01:01:22.655319 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.655296    4327 scope.go:122] "RemoveContainer" containerID="b3268d824f6459a5ce45f602b372e77676b9abc1eee0ed4dc6232c980ca0d3bf"
Oct 31 01:01:22.684105 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.684073    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"]
Oct 31 01:01:22.696468 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.696432    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hk9qf"]
Oct 31 01:01:22.722289 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.722259    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"]
Oct 31 01:01:22.726481 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.726454    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4qhrd"]
Oct 31 01:01:22.767450 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.767416    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"]
Oct 31 01:01:22.773822 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:22.773796    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8zxnz"]
Oct 31 01:01:23.945122 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:23.944369    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3" path="/var/lib/kubelet/pods/6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3/volumes"
Oct 31 01:01:23.945651 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:23.945192    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="820c95b2-cfd6-4893-80ed-1c746d4cfef2" path="/var/lib/kubelet/pods/820c95b2-cfd6-4893-80ed-1c746d4cfef2/volumes"
Oct 31 01:01:23.945949 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:23.945917    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f7f289-ef35-494d-91c9-034c7f1294d4" path="/var/lib/kubelet/pods/a5f7f289-ef35-494d-91c9-034c7f1294d4/volumes"
Oct 31 01:01:29.527565 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:29.527519    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9" podUID="67570d88-4047-46e3-9c60-3522e8c59493" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:01:29.528332 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:29.528305    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd" podUID="d628dce5-6346-46cc-a356-ad526ffd0b96" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:01:30.018524 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.018488    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd"
Oct 31 01:01:30.054808 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.054781    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9"
Oct 31 01:01:30.085203 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.085157    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d628dce5-6346-46cc-a356-ad526ffd0b96-kube-api-access-tlzgd\" (UniqueName: \"kubernetes.io/projected/d628dce5-6346-46cc-a356-ad526ffd0b96-kube-api-access-tlzgd\") pod \"d628dce5-6346-46cc-a356-ad526ffd0b96\" (UID: \"d628dce5-6346-46cc-a356-ad526ffd0b96\") "
Oct 31 01:01:30.085402 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.085357    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d628dce5-6346-46cc-a356-ad526ffd0b96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d628dce5-6346-46cc-a356-ad526ffd0b96-plugins-config-volume\") pod \"d628dce5-6346-46cc-a356-ad526ffd0b96\" (UID: \"d628dce5-6346-46cc-a356-ad526ffd0b96\") "
Oct 31 01:01:30.085552 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.085530    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67570d88-4047-46e3-9c60-3522e8c59493-kube-api-access-qbtzz\" (UniqueName: \"kubernetes.io/projected/67570d88-4047-46e3-9c60-3522e8c59493-kube-api-access-qbtzz\") pod \"67570d88-4047-46e3-9c60-3522e8c59493\" (UID: \"67570d88-4047-46e3-9c60-3522e8c59493\") "
Oct 31 01:01:30.085701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.085559    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67570d88-4047-46e3-9c60-3522e8c59493-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67570d88-4047-46e3-9c60-3522e8c59493-plugins-config-volume\") pod \"67570d88-4047-46e3-9c60-3522e8c59493\" (UID: \"67570d88-4047-46e3-9c60-3522e8c59493\") "
Oct 31 01:01:30.086018 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.085991    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d628dce5-6346-46cc-a356-ad526ffd0b96-plugins-config-volume" pod "d628dce5-6346-46cc-a356-ad526ffd0b96" (UID: "d628dce5-6346-46cc-a356-ad526ffd0b96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:01:30.086117 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.086066    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67570d88-4047-46e3-9c60-3522e8c59493-plugins-config-volume" pod "67570d88-4047-46e3-9c60-3522e8c59493" (UID: "67570d88-4047-46e3-9c60-3522e8c59493"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:01:30.086451 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.086428    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d628dce5-6346-46cc-a356-ad526ffd0b96-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:30.086496 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.086459    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67570d88-4047-46e3-9c60-3522e8c59493-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:30.087486 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.087459    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d628dce5-6346-46cc-a356-ad526ffd0b96-kube-api-access-tlzgd" pod "d628dce5-6346-46cc-a356-ad526ffd0b96" (UID: "d628dce5-6346-46cc-a356-ad526ffd0b96"). InnerVolumeSpecName "kube-api-access-tlzgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:01:30.087824 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.087800    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67570d88-4047-46e3-9c60-3522e8c59493-kube-api-access-qbtzz" pod "67570d88-4047-46e3-9c60-3522e8c59493" (UID: "67570d88-4047-46e3-9c60-3522e8c59493"). InnerVolumeSpecName "kube-api-access-qbtzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:01:30.187719 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.187622    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlzgd\" (UniqueName: \"kubernetes.io/projected/d628dce5-6346-46cc-a356-ad526ffd0b96-kube-api-access-tlzgd\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:30.187719 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.187652    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbtzz\" (UniqueName: \"kubernetes.io/projected/67570d88-4047-46e3-9c60-3522e8c59493-kube-api-access-qbtzz\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:30.227805 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.227768    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc" podUID="841015e0-30ae-4810-a913-598be430b8fd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:01:30.313398 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.313357    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28" podUID="f113a57a-325b-4508-ad0f-3e9677b3f95d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:01:30.699986 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.699943    4327 generic.go:356] "Generic (PLEG): container finished" podID="d628dce5-6346-46cc-a356-ad526ffd0b96" containerID="d77a8c61c24b16dbc4269489c7d97b64eacdff26c7c1922c2043e0633389359c" exitCode=0
Oct 31 01:01:30.700446 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.699993    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd"
Oct 31 01:01:30.700446 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.700024    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd" event={"ID":"d628dce5-6346-46cc-a356-ad526ffd0b96","Type":"ContainerDied","Data":"d77a8c61c24b16dbc4269489c7d97b64eacdff26c7c1922c2043e0633389359c"}
Oct 31 01:01:30.700446 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.700078    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd" event={"ID":"d628dce5-6346-46cc-a356-ad526ffd0b96","Type":"ContainerDied","Data":"312ab59c37e3a931ee09cf11cedf3a064896aae89ebeaae7eb41da7addbfdac6"}
Oct 31 01:01:30.700446 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.700100    4327 scope.go:122] "RemoveContainer" containerID="d77a8c61c24b16dbc4269489c7d97b64eacdff26c7c1922c2043e0633389359c"
Oct 31 01:01:30.701765 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.701737    4327 generic.go:356] "Generic (PLEG): container finished" podID="67570d88-4047-46e3-9c60-3522e8c59493" containerID="a54e5003fd0d96d98aca740d43e27a7c303bd014adf735ec1b9e5370510b72bc" exitCode=0
Oct 31 01:01:30.701888 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.701788    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9"
Oct 31 01:01:30.701888 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.701823    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9" event={"ID":"67570d88-4047-46e3-9c60-3522e8c59493","Type":"ContainerDied","Data":"a54e5003fd0d96d98aca740d43e27a7c303bd014adf735ec1b9e5370510b72bc"}
Oct 31 01:01:30.701888 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.701879    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9" event={"ID":"67570d88-4047-46e3-9c60-3522e8c59493","Type":"ContainerDied","Data":"ac2a905f0d575930b2337d4798a06e21ac664e811ce35294ff86dd3c21af68e9"}
Oct 31 01:01:30.708126 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.708100    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"
Oct 31 01:01:30.713268 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.713246    4327 scope.go:122] "RemoveContainer" containerID="d77a8c61c24b16dbc4269489c7d97b64eacdff26c7c1922c2043e0633389359c"
Oct 31 01:01:30.713606 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:01:30.713577    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d77a8c61c24b16dbc4269489c7d97b64eacdff26c7c1922c2043e0633389359c\": container with ID starting with d77a8c61c24b16dbc4269489c7d97b64eacdff26c7c1922c2043e0633389359c not found: ID does not exist" containerID="d77a8c61c24b16dbc4269489c7d97b64eacdff26c7c1922c2043e0633389359c"
Oct 31 01:01:30.713724 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.713615    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d77a8c61c24b16dbc4269489c7d97b64eacdff26c7c1922c2043e0633389359c"} err="failed to get container status \"d77a8c61c24b16dbc4269489c7d97b64eacdff26c7c1922c2043e0633389359c\": rpc error: code = NotFound desc = could not find container \"d77a8c61c24b16dbc4269489c7d97b64eacdff26c7c1922c2043e0633389359c\": container with ID starting with d77a8c61c24b16dbc4269489c7d97b64eacdff26c7c1922c2043e0633389359c not found: ID does not exist"
Oct 31 01:01:30.713724 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.713635    4327 scope.go:122] "RemoveContainer" containerID="a54e5003fd0d96d98aca740d43e27a7c303bd014adf735ec1b9e5370510b72bc"
Oct 31 01:01:30.722447 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.722428    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"
Oct 31 01:01:30.724356 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.724336    4327 scope.go:122] "RemoveContainer" containerID="a54e5003fd0d96d98aca740d43e27a7c303bd014adf735ec1b9e5370510b72bc"
Oct 31 01:01:30.724646 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:01:30.724629    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a54e5003fd0d96d98aca740d43e27a7c303bd014adf735ec1b9e5370510b72bc\": container with ID starting with a54e5003fd0d96d98aca740d43e27a7c303bd014adf735ec1b9e5370510b72bc not found: ID does not exist" containerID="a54e5003fd0d96d98aca740d43e27a7c303bd014adf735ec1b9e5370510b72bc"
Oct 31 01:01:30.724729 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.724653    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54e5003fd0d96d98aca740d43e27a7c303bd014adf735ec1b9e5370510b72bc"} err="failed to get container status \"a54e5003fd0d96d98aca740d43e27a7c303bd014adf735ec1b9e5370510b72bc\": rpc error: code = NotFound desc = could not find container \"a54e5003fd0d96d98aca740d43e27a7c303bd014adf735ec1b9e5370510b72bc\": container with ID starting with a54e5003fd0d96d98aca740d43e27a7c303bd014adf735ec1b9e5370510b72bc not found: ID does not exist"
Oct 31 01:01:30.754801 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.754764    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd"]
Oct 31 01:01:30.768277 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.768247    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c6d97d-68bd6f89-nwznd"]
Oct 31 01:01:30.796690 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.796639    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9"]
Oct 31 01:01:30.800463 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:30.800440    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c6d97d-59565cfc55-pkzq9"]
Oct 31 01:01:31.720205 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:31.720175    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"
Oct 31 01:01:31.924049 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:31.924015    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67570d88-4047-46e3-9c60-3522e8c59493" path="/var/lib/kubelet/pods/67570d88-4047-46e3-9c60-3522e8c59493/volumes"
Oct 31 01:01:31.924512 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:31.924487    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d628dce5-6346-46cc-a356-ad526ffd0b96" path="/var/lib/kubelet/pods/d628dce5-6346-46cc-a356-ad526ffd0b96/volumes"
Oct 31 01:01:36.994025 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:36.993558    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc"
Oct 31 01:01:37.097729 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.097674    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28"
Oct 31 01:01:37.100765 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.100746    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/841015e0-30ae-4810-a913-598be430b8fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841015e0-30ae-4810-a913-598be430b8fd-plugins-config-volume\") pod \"841015e0-30ae-4810-a913-598be430b8fd\" (UID: \"841015e0-30ae-4810-a913-598be430b8fd\") "
Oct 31 01:01:37.100958 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.100944    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/841015e0-30ae-4810-a913-598be430b8fd-kube-api-access-tdp7w\" (UniqueName: \"kubernetes.io/projected/841015e0-30ae-4810-a913-598be430b8fd-kube-api-access-tdp7w\") pod \"841015e0-30ae-4810-a913-598be430b8fd\" (UID: \"841015e0-30ae-4810-a913-598be430b8fd\") "
Oct 31 01:01:37.101137 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.101112    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841015e0-30ae-4810-a913-598be430b8fd-plugins-config-volume" pod "841015e0-30ae-4810-a913-598be430b8fd" (UID: "841015e0-30ae-4810-a913-598be430b8fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:01:37.101447 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.101426    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841015e0-30ae-4810-a913-598be430b8fd-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:37.103104 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.103078    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/841015e0-30ae-4810-a913-598be430b8fd-kube-api-access-tdp7w" pod "841015e0-30ae-4810-a913-598be430b8fd" (UID: "841015e0-30ae-4810-a913-598be430b8fd"). InnerVolumeSpecName "kube-api-access-tdp7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:01:37.202942 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.202908    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f113a57a-325b-4508-ad0f-3e9677b3f95d-kube-api-access-q8pbk\" (UniqueName: \"kubernetes.io/projected/f113a57a-325b-4508-ad0f-3e9677b3f95d-kube-api-access-q8pbk\") pod \"f113a57a-325b-4508-ad0f-3e9677b3f95d\" (UID: \"f113a57a-325b-4508-ad0f-3e9677b3f95d\") "
Oct 31 01:01:37.203126 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.203076    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f113a57a-325b-4508-ad0f-3e9677b3f95d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f113a57a-325b-4508-ad0f-3e9677b3f95d-plugins-config-volume\") pod \"f113a57a-325b-4508-ad0f-3e9677b3f95d\" (UID: \"f113a57a-325b-4508-ad0f-3e9677b3f95d\") "
Oct 31 01:01:37.203453 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.203426    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f113a57a-325b-4508-ad0f-3e9677b3f95d-plugins-config-volume" pod "f113a57a-325b-4508-ad0f-3e9677b3f95d" (UID: "f113a57a-325b-4508-ad0f-3e9677b3f95d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:01:37.203550 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.203484    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdp7w\" (UniqueName: \"kubernetes.io/projected/841015e0-30ae-4810-a913-598be430b8fd-kube-api-access-tdp7w\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:37.205217 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.205194    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f113a57a-325b-4508-ad0f-3e9677b3f95d-kube-api-access-q8pbk" pod "f113a57a-325b-4508-ad0f-3e9677b3f95d" (UID: "f113a57a-325b-4508-ad0f-3e9677b3f95d"). InnerVolumeSpecName "kube-api-access-q8pbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:01:37.305162 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.305119    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8pbk\" (UniqueName: \"kubernetes.io/projected/f113a57a-325b-4508-ad0f-3e9677b3f95d-kube-api-access-q8pbk\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:37.305162 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.305156    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f113a57a-325b-4508-ad0f-3e9677b3f95d-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:37.764316 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.764279    4327 generic.go:356] "Generic (PLEG): container finished" podID="f113a57a-325b-4508-ad0f-3e9677b3f95d" containerID="d2b98a8a41303c17b12e9bdfb78a79c9bb640cbf7d68c8354e1dd9c32104d5b1" exitCode=0
Oct 31 01:01:37.764491 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.764322    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28"
Oct 31 01:01:37.764491 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.764368    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28" event={"ID":"f113a57a-325b-4508-ad0f-3e9677b3f95d","Type":"ContainerDied","Data":"d2b98a8a41303c17b12e9bdfb78a79c9bb640cbf7d68c8354e1dd9c32104d5b1"}
Oct 31 01:01:37.764491 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.764419    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28" event={"ID":"f113a57a-325b-4508-ad0f-3e9677b3f95d","Type":"ContainerDied","Data":"fc449ea5a7f3d14c49262525a0fc4047c1c6241268f25cc509b9ee3aecf88591"}
Oct 31 01:01:37.764491 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.764484    4327 scope.go:122] "RemoveContainer" containerID="d2b98a8a41303c17b12e9bdfb78a79c9bb640cbf7d68c8354e1dd9c32104d5b1"
Oct 31 01:01:37.766435 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.766401    4327 generic.go:356] "Generic (PLEG): container finished" podID="841015e0-30ae-4810-a913-598be430b8fd" containerID="a1f7c1812ca2d57653d4c723de70bff882d8f7d43871abd93487ddd144887b20" exitCode=0
Oct 31 01:01:37.766543 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.766459    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc"
Oct 31 01:01:37.766543 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.766459    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc" event={"ID":"841015e0-30ae-4810-a913-598be430b8fd","Type":"ContainerDied","Data":"a1f7c1812ca2d57653d4c723de70bff882d8f7d43871abd93487ddd144887b20"}
Oct 31 01:01:37.766543 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.766514    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc" event={"ID":"841015e0-30ae-4810-a913-598be430b8fd","Type":"ContainerDied","Data":"a77450732a1516b64afeee1fbea07dbce84c3c3229866266f3cfb3ce45880c9c"}
Oct 31 01:01:37.780237 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.780212    4327 scope.go:122] "RemoveContainer" containerID="d2b98a8a41303c17b12e9bdfb78a79c9bb640cbf7d68c8354e1dd9c32104d5b1"
Oct 31 01:01:37.780922 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:01:37.780892    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2b98a8a41303c17b12e9bdfb78a79c9bb640cbf7d68c8354e1dd9c32104d5b1\": container with ID starting with d2b98a8a41303c17b12e9bdfb78a79c9bb640cbf7d68c8354e1dd9c32104d5b1 not found: ID does not exist" containerID="d2b98a8a41303c17b12e9bdfb78a79c9bb640cbf7d68c8354e1dd9c32104d5b1"
Oct 31 01:01:37.781042 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.780943    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b98a8a41303c17b12e9bdfb78a79c9bb640cbf7d68c8354e1dd9c32104d5b1"} err="failed to get container status \"d2b98a8a41303c17b12e9bdfb78a79c9bb640cbf7d68c8354e1dd9c32104d5b1\": rpc error: code = NotFound desc = could not find container \"d2b98a8a41303c17b12e9bdfb78a79c9bb640cbf7d68c8354e1dd9c32104d5b1\": container with ID starting with d2b98a8a41303c17b12e9bdfb78a79c9bb640cbf7d68c8354e1dd9c32104d5b1 not found: ID does not exist"
Oct 31 01:01:37.781042 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.780967    4327 scope.go:122] "RemoveContainer" containerID="a1f7c1812ca2d57653d4c723de70bff882d8f7d43871abd93487ddd144887b20"
Oct 31 01:01:37.792412 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.792393    4327 scope.go:122] "RemoveContainer" containerID="a1f7c1812ca2d57653d4c723de70bff882d8f7d43871abd93487ddd144887b20"
Oct 31 01:01:37.792754 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:01:37.792732    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f7c1812ca2d57653d4c723de70bff882d8f7d43871abd93487ddd144887b20\": container with ID starting with a1f7c1812ca2d57653d4c723de70bff882d8f7d43871abd93487ddd144887b20 not found: ID does not exist" containerID="a1f7c1812ca2d57653d4c723de70bff882d8f7d43871abd93487ddd144887b20"
Oct 31 01:01:37.792805 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.792769    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f7c1812ca2d57653d4c723de70bff882d8f7d43871abd93487ddd144887b20"} err="failed to get container status \"a1f7c1812ca2d57653d4c723de70bff882d8f7d43871abd93487ddd144887b20\": rpc error: code = NotFound desc = could not find container \"a1f7c1812ca2d57653d4c723de70bff882d8f7d43871abd93487ddd144887b20\": container with ID starting with a1f7c1812ca2d57653d4c723de70bff882d8f7d43871abd93487ddd144887b20 not found: ID does not exist"
Oct 31 01:01:37.819993 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.819963    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28"]
Oct 31 01:01:37.823864 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.823834    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fd1dfc-5fc4c54799-65s28"]
Oct 31 01:01:37.872086 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.872052    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc"]
Oct 31 01:01:37.877666 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.877640    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fd1dfc-6b7b57d44f-9p9pc"]
Oct 31 01:01:37.923867 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.923826    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="841015e0-30ae-4810-a913-598be430b8fd" path="/var/lib/kubelet/pods/841015e0-30ae-4810-a913-598be430b8fd/volumes"
Oct 31 01:01:37.924461 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:37.924443    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f113a57a-325b-4508-ad0f-3e9677b3f95d" path="/var/lib/kubelet/pods/f113a57a-325b-4508-ad0f-3e9677b3f95d/volumes"
Oct 31 01:01:41.382700 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:41.382612    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv"]
Oct 31 01:01:41.383222 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:41.382913    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv" podUID="75942e65-46b1-4ec3-84a7-425e74bd1296" containerName="maas-api" containerID="cri-o://24b73a577c0306d7e800a374e43f612308b69b4c9df9e294aeb415d12148306b" gracePeriod=30
Oct 31 01:01:41.960888 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:41.960796    4327 generic.go:356] "Generic (PLEG): container finished" podID="75942e65-46b1-4ec3-84a7-425e74bd1296" containerID="24b73a577c0306d7e800a374e43f612308b69b4c9df9e294aeb415d12148306b" exitCode=0
Oct 31 01:01:41.975851 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:41.973854    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv" event={"ID":"75942e65-46b1-4ec3-84a7-425e74bd1296","Type":"ContainerDied","Data":"24b73a577c0306d7e800a374e43f612308b69b4c9df9e294aeb415d12148306b"}
Oct 31 01:01:42.067513 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:42.067483    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv"
Oct 31 01:01:42.125720 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:42.125673    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75942e65-46b1-4ec3-84a7-425e74bd1296-metrics-certs\" (UniqueName: \"kubernetes.io/secret/75942e65-46b1-4ec3-84a7-425e74bd1296-metrics-certs\") pod \"75942e65-46b1-4ec3-84a7-425e74bd1296\" (UID: \"75942e65-46b1-4ec3-84a7-425e74bd1296\") "
Oct 31 01:01:42.125946 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:42.125783    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75942e65-46b1-4ec3-84a7-425e74bd1296-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75942e65-46b1-4ec3-84a7-425e74bd1296-openshift-service-ca\") pod \"75942e65-46b1-4ec3-84a7-425e74bd1296\" (UID: \"75942e65-46b1-4ec3-84a7-425e74bd1296\") "
Oct 31 01:01:42.126048 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:42.126012    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75942e65-46b1-4ec3-84a7-425e74bd1296-kube-api-access-85wp9\" (UniqueName: \"kubernetes.io/projected/75942e65-46b1-4ec3-84a7-425e74bd1296-kube-api-access-85wp9\") pod \"75942e65-46b1-4ec3-84a7-425e74bd1296\" (UID: \"75942e65-46b1-4ec3-84a7-425e74bd1296\") "
Oct 31 01:01:42.126114 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:42.126056    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75942e65-46b1-4ec3-84a7-425e74bd1296-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75942e65-46b1-4ec3-84a7-425e74bd1296-maas-api-tls\") pod \"75942e65-46b1-4ec3-84a7-425e74bd1296\" (UID: \"75942e65-46b1-4ec3-84a7-425e74bd1296\") "
Oct 31 01:01:42.126276 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:42.126210    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75942e65-46b1-4ec3-84a7-425e74bd1296-openshift-service-ca" pod "75942e65-46b1-4ec3-84a7-425e74bd1296" (UID: "75942e65-46b1-4ec3-84a7-425e74bd1296"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:01:42.126726 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:42.126646    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75942e65-46b1-4ec3-84a7-425e74bd1296-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:42.131990 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:42.131959    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75942e65-46b1-4ec3-84a7-425e74bd1296-metrics-certs" pod "75942e65-46b1-4ec3-84a7-425e74bd1296" (UID: "75942e65-46b1-4ec3-84a7-425e74bd1296"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:01:42.132334 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:42.132308    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75942e65-46b1-4ec3-84a7-425e74bd1296-kube-api-access-85wp9" pod "75942e65-46b1-4ec3-84a7-425e74bd1296" (UID: "75942e65-46b1-4ec3-84a7-425e74bd1296"). InnerVolumeSpecName "kube-api-access-85wp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:01:42.134335 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:42.134307    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75942e65-46b1-4ec3-84a7-425e74bd1296-maas-api-tls" pod "75942e65-46b1-4ec3-84a7-425e74bd1296" (UID: "75942e65-46b1-4ec3-84a7-425e74bd1296"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:01:42.228380 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:42.228339    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85wp9\" (UniqueName: \"kubernetes.io/projected/75942e65-46b1-4ec3-84a7-425e74bd1296-kube-api-access-85wp9\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:42.228380 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:42.228377    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75942e65-46b1-4ec3-84a7-425e74bd1296-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:42.228641 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:42.228390    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75942e65-46b1-4ec3-84a7-425e74bd1296-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:42.970849 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:42.970820    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv"
Oct 31 01:01:42.971295 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:42.970846    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv" event={"ID":"75942e65-46b1-4ec3-84a7-425e74bd1296","Type":"ContainerDied","Data":"68e901a237fb79541d3d9b7bca83536ad372e3fe246cbc617b62e8226bd02083"}
Oct 31 01:01:42.971295 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:42.970913    4327 scope.go:122] "RemoveContainer" containerID="24b73a577c0306d7e800a374e43f612308b69b4c9df9e294aeb415d12148306b"
Oct 31 01:01:43.035049 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:43.035018    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv"]
Oct 31 01:01:43.047018 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:43.046990    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd1dfc-669c87c448-vxztv"]
Oct 31 01:01:43.924244 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:43.924209    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75942e65-46b1-4ec3-84a7-425e74bd1296" path="/var/lib/kubelet/pods/75942e65-46b1-4ec3-84a7-425e74bd1296/volumes"
Oct 31 01:01:45.294614 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:01:45.294576    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:01:50.943930 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:50.943894    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"]
Oct 31 01:01:50.944371 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:50.944118    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d" podUID="79f228f5-1947-426c-9e4d-4f300f43511e" containerName="istio-proxy" containerID="cri-o://1c7268daabd20de5cb1fc2b74dab2150c455e7a3a1d83323d5c65bc2b519388e" gracePeriod=30
Oct 31 01:01:51.304641 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:51.304543    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"]
Oct 31 01:01:51.304865 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:51.304835    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv" podUID="47727a40-911f-4d8d-8fd6-f19ada8e4ea0" containerName="istio-proxy" containerID="cri-o://b94f41e5ac21207e05812b0614472387e51061e4bd6c94b34a6a2e58c681779f" gracePeriod=30
Oct 31 01:01:52.358543 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.358505    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"]
Oct 31 01:01:52.360167 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360141    4327 cpu_manager.go:414] "containerMap: removing container" podUID="841015e0-30ae-4810-a913-598be430b8fd" containerName="payload-processing"
Oct 31 01:01:52.360167 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360165    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="841015e0-30ae-4810-a913-598be430b8fd" containerName="payload-processing"
Oct 31 01:01:52.360357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360184    4327 cpu_manager.go:414] "containerMap: removing container" podUID="f45f589e-69d1-49fd-b27d-a15d1f886235" containerName="cleanup"
Oct 31 01:01:52.360357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360192    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45f589e-69d1-49fd-b27d-a15d1f886235" containerName="cleanup"
Oct 31 01:01:52.360357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360205    4327 cpu_manager.go:414] "containerMap: removing container" podUID="6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3" containerName="istio-proxy"
Oct 31 01:01:52.360357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360212    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3" containerName="istio-proxy"
Oct 31 01:01:52.360357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360228    4327 cpu_manager.go:414] "containerMap: removing container" podUID="f113a57a-325b-4508-ad0f-3e9677b3f95d" containerName="payload-pre-processing"
Oct 31 01:01:52.360357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360235    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f113a57a-325b-4508-ad0f-3e9677b3f95d" containerName="payload-pre-processing"
Oct 31 01:01:52.360357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360254    4327 cpu_manager.go:414] "containerMap: removing container" podUID="b118e1e7-3dd9-4107-b740-9032c60dee40" containerName="cleanup"
Oct 31 01:01:52.360357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360263    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b118e1e7-3dd9-4107-b740-9032c60dee40" containerName="cleanup"
Oct 31 01:01:52.360357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360280    4327 cpu_manager.go:414] "containerMap: removing container" podUID="d628dce5-6346-46cc-a356-ad526ffd0b96" containerName="payload-processing"
Oct 31 01:01:52.360357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360287    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d628dce5-6346-46cc-a356-ad526ffd0b96" containerName="payload-processing"
Oct 31 01:01:52.360357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360299    4327 cpu_manager.go:414] "containerMap: removing container" podUID="75942e65-46b1-4ec3-84a7-425e74bd1296" containerName="maas-api"
Oct 31 01:01:52.360357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360306    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75942e65-46b1-4ec3-84a7-425e74bd1296" containerName="maas-api"
Oct 31 01:01:52.360357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360340    4327 cpu_manager.go:414] "containerMap: removing container" podUID="c4ee6cb0-2737-426b-9f00-4d9f550c8164" containerName="cleanup"
Oct 31 01:01:52.360357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360347    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ee6cb0-2737-426b-9f00-4d9f550c8164" containerName="cleanup"
Oct 31 01:01:52.360357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360361    4327 cpu_manager.go:414] "containerMap: removing container" podUID="67570d88-4047-46e3-9c60-3522e8c59493" containerName="payload-pre-processing"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360369    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67570d88-4047-46e3-9c60-3522e8c59493" containerName="payload-pre-processing"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360381    4327 cpu_manager.go:414] "containerMap: removing container" podUID="a5f7f289-ef35-494d-91c9-034c7f1294d4" containerName="istio-proxy"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360388    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f7f289-ef35-494d-91c9-034c7f1294d4" containerName="istio-proxy"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360408    4327 cpu_manager.go:414] "containerMap: removing container" podUID="d352532d-5b9b-4713-a13c-32cd6d2c7716" containerName="cleanup"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360414    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d352532d-5b9b-4713-a13c-32cd6d2c7716" containerName="cleanup"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360436    4327 cpu_manager.go:414] "containerMap: removing container" podUID="8b6d9a4f-7b06-42f1-8e32-720489b7ba93" containerName="cleanup"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360444    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b6d9a4f-7b06-42f1-8e32-720489b7ba93" containerName="cleanup"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360459    4327 cpu_manager.go:414] "containerMap: removing container" podUID="13752887-b28f-49f8-855d-2eed4657dc61" containerName="cleanup"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360466    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13752887-b28f-49f8-855d-2eed4657dc61" containerName="cleanup"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360480    4327 cpu_manager.go:414] "containerMap: removing container" podUID="11188116-a59b-48f1-9f7d-85d09032455e" containerName="cleanup"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360487    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11188116-a59b-48f1-9f7d-85d09032455e" containerName="cleanup"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360505    4327 cpu_manager.go:414] "containerMap: removing container" podUID="7f085592-c63f-433e-a77f-0b0c20746fe2" containerName="cleanup"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360512    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f085592-c63f-433e-a77f-0b0c20746fe2" containerName="cleanup"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360533    4327 cpu_manager.go:414] "containerMap: removing container" podUID="820c95b2-cfd6-4893-80ed-1c746d4cfef2" containerName="istio-proxy"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360540    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="820c95b2-cfd6-4893-80ed-1c746d4cfef2" containerName="istio-proxy"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360836    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="820c95b2-cfd6-4893-80ed-1c746d4cfef2" containerName="istio-proxy"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360861    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="841015e0-30ae-4810-a913-598be430b8fd" containerName="payload-processing"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360876    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45f589e-69d1-49fd-b27d-a15d1f886235" containerName="cleanup"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360888    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="6af61687-4a18-4cf5-85e0-aa2ae8a2a2d3" containerName="istio-proxy"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360903    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="f113a57a-325b-4508-ad0f-3e9677b3f95d" containerName="payload-pre-processing"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360914    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="b118e1e7-3dd9-4107-b740-9032c60dee40" containerName="cleanup"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360927    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="d628dce5-6346-46cc-a356-ad526ffd0b96" containerName="payload-processing"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360937    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="75942e65-46b1-4ec3-84a7-425e74bd1296" containerName="maas-api"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360956    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="67570d88-4047-46e3-9c60-3522e8c59493" containerName="payload-pre-processing"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360967    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f7f289-ef35-494d-91c9-034c7f1294d4" containerName="istio-proxy"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360978    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="d352532d-5b9b-4713-a13c-32cd6d2c7716" containerName="cleanup"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.360991    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f085592-c63f-433e-a77f-0b0c20746fe2" containerName="cleanup"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.361001    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="11188116-a59b-48f1-9f7d-85d09032455e" containerName="cleanup"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.361013    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="e30c1c23-92c5-47b6-a250-3519dbfae94b" containerName="istio-proxy"
Oct 31 01:01:52.361168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.361023    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="13752887-b28f-49f8-855d-2eed4657dc61" containerName="cleanup"
Oct 31 01:01:52.367089 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.367067    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.397543 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.396356    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-476a67-openshift-default-dockercfg-nc8cg\""
Oct 31 01:01:52.404028 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.403996    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"]
Oct 31 01:01:52.424327 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.424293    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-credential-socket\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.424539 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.424405    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-workload-socket\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.424539 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.424502    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-data\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.424744 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.424553    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5lbw\" (UniqueName: \"kubernetes.io/projected/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-kube-api-access-j5lbw\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.426374 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.424864    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-envoy\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.426374 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.425025    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-podinfo\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.426374 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.425150    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istiod-ca-cert\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.426374 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.425226    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-workload-certs\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.426374 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.425279    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-token\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.577437 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.544985    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-envoy\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.614382 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.592203    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-podinfo\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.614382 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.592341    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istiod-ca-cert\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.614382 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.592406    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-workload-certs\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.614382 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.592434    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-token\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.614382 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.592484    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-credential-socket\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.614382 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.592559    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-workload-socket\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.614382 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.592651    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-data\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.614382 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.592737    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5lbw\" (UniqueName: \"kubernetes.io/projected/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-kube-api-access-j5lbw\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.614382 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.596817    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-podinfo\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.614382 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.597469    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istiod-ca-cert\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.614382 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.599131    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-workload-certs\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.614382 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.599640    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-credential-socket\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.614382 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.600050    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-workload-socket\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.614382 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.600291    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-data\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.614382 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.602281    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-envoy\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.627236 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.627164    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5lbw\" (UniqueName: \"kubernetes.io/projected/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-kube-api-access-j5lbw\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.632261 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.632195    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-token\") pod \"e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") " pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:52.685196 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:52.685161    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:53.030561 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:01:53.030534    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ff4194_5ef3_4f4d_b6e7_4cb5df757371.slice/crio-33d78ee06f6b194d45b125df8ad485b09427a80b71517e2dbd9cc04bb39e106b WatchSource:0}: Error finding container 33d78ee06f6b194d45b125df8ad485b09427a80b71517e2dbd9cc04bb39e106b: Status 404 returned error can't find the container with id 33d78ee06f6b194d45b125df8ad485b09427a80b71517e2dbd9cc04bb39e106b
Oct 31 01:01:53.031763 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:53.031741    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"]
Oct 31 01:01:53.033890 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:53.033855    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:01:53.033962 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:53.033924    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:01:53.033962 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:53.033954    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:01:53.077828 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:53.077796    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz" event={"ID":"13ff4194-5ef3-4f4d-b6e7-4cb5df757371","Type":"ContainerStarted","Data":"33d78ee06f6b194d45b125df8ad485b09427a80b71517e2dbd9cc04bb39e106b"}
Oct 31 01:01:54.088051 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:54.088013    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz" event={"ID":"13ff4194-5ef3-4f4d-b6e7-4cb5df757371","Type":"ContainerStarted","Data":"fb69bc630de7bea5dd4698a8804940d4e0b4314d8bba5a78937eabfc882e2f05"}
Oct 31 01:01:54.166039 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:54.165977    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz" podStartSLOduration=2.165963964 podStartE2EDuration="2.165963964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:01:54.153559985 +0000 UTC m=+2574.375595528" watchObservedRunningTime="2026-10-31 01:01:54.165963964 +0000 UTC m=+2574.387999505"
Oct 31 01:01:54.685458 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:54.685418    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:54.688786 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:54.688754    4327 patch_prober.go:29] interesting pod/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:01:54.688963 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:54.688806    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d" podUID="79f228f5-1947-426c-9e4d-4f300f43511e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:01:54.691852 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:54.691825    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:55.095817 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:55.095728    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:55.096864 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:55.096839    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:01:56.107744 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.107612    4327 generic.go:356] "Generic (PLEG): container finished" podID="79f228f5-1947-426c-9e4d-4f300f43511e" containerID="1c7268daabd20de5cb1fc2b74dab2150c455e7a3a1d83323d5c65bc2b519388e" exitCode=0
Oct 31 01:01:56.108272 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.107793    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d" event={"ID":"79f228f5-1947-426c-9e4d-4f300f43511e","Type":"ContainerDied","Data":"1c7268daabd20de5cb1fc2b74dab2150c455e7a3a1d83323d5c65bc2b519388e"}
Oct 31 01:01:56.108272 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.107893    4327 scope.go:122] "RemoveContainer" containerID="6a50b7d905e033190187c39065ee1df2811ab7c476e902c359e8fdedd4b5b8ea"
Oct 31 01:01:56.236933 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.236902    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 01:01:56.359541 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.359493    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79f228f5-1947-426c-9e4d-4f300f43511e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79f228f5-1947-426c-9e4d-4f300f43511e-istio-podinfo\") pod \"79f228f5-1947-426c-9e4d-4f300f43511e\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") "
Oct 31 01:01:56.359693 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.359564    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-istio-data\") pod \"79f228f5-1947-426c-9e4d-4f300f43511e\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") "
Oct 31 01:01:56.359693 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.359602    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-istio-envoy\") pod \"79f228f5-1947-426c-9e4d-4f300f43511e\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") "
Oct 31 01:01:56.359693 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.359673    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79f228f5-1947-426c-9e4d-4f300f43511e-kube-api-access-sg7cz\" (UniqueName: \"kubernetes.io/projected/79f228f5-1947-426c-9e4d-4f300f43511e-kube-api-access-sg7cz\") pod \"79f228f5-1947-426c-9e4d-4f300f43511e\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") "
Oct 31 01:01:56.359893 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.359874    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79f228f5-1947-426c-9e4d-4f300f43511e-istio-token\" (UniqueName: \"kubernetes.io/projected/79f228f5-1947-426c-9e4d-4f300f43511e-istio-token\") pod \"79f228f5-1947-426c-9e4d-4f300f43511e\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") "
Oct 31 01:01:56.359990 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.359974    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-credential-socket\") pod \"79f228f5-1947-426c-9e4d-4f300f43511e\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") "
Oct 31 01:01:56.360049 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.360031    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-workload-socket\") pod \"79f228f5-1947-426c-9e4d-4f300f43511e\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") "
Oct 31 01:01:56.360372 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.360111    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79f228f5-1947-426c-9e4d-4f300f43511e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79f228f5-1947-426c-9e4d-4f300f43511e-istiod-ca-cert\") pod \"79f228f5-1947-426c-9e4d-4f300f43511e\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") "
Oct 31 01:01:56.360372 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.360178    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-workload-certs\") pod \"79f228f5-1947-426c-9e4d-4f300f43511e\" (UID: \"79f228f5-1947-426c-9e4d-4f300f43511e\") "
Oct 31 01:01:56.360987 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.360775    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-workload-socket" pod "79f228f5-1947-426c-9e4d-4f300f43511e" (UID: "79f228f5-1947-426c-9e4d-4f300f43511e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:56.361556 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.361038    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-istio-data" pod "79f228f5-1947-426c-9e4d-4f300f43511e" (UID: "79f228f5-1947-426c-9e4d-4f300f43511e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:56.361556 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.361281    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-workload-certs" pod "79f228f5-1947-426c-9e4d-4f300f43511e" (UID: "79f228f5-1947-426c-9e4d-4f300f43511e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:56.361556 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.361473    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f228f5-1947-426c-9e4d-4f300f43511e-istiod-ca-cert" pod "79f228f5-1947-426c-9e4d-4f300f43511e" (UID: "79f228f5-1947-426c-9e4d-4f300f43511e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:01:56.361961 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.361811    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-credential-socket" pod "79f228f5-1947-426c-9e4d-4f300f43511e" (UID: "79f228f5-1947-426c-9e4d-4f300f43511e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:56.362472 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.362378    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:56.362472 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.362406    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:56.362472 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.362424    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:56.362472 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.362439    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79f228f5-1947-426c-9e4d-4f300f43511e-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:56.362472 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.362454    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:56.363029 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.363002    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79f228f5-1947-426c-9e4d-4f300f43511e-istio-podinfo" pod "79f228f5-1947-426c-9e4d-4f300f43511e" (UID: "79f228f5-1947-426c-9e4d-4f300f43511e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:01:56.363408 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.363381    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-istio-envoy" pod "79f228f5-1947-426c-9e4d-4f300f43511e" (UID: "79f228f5-1947-426c-9e4d-4f300f43511e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:56.364022 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.363997    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f228f5-1947-426c-9e4d-4f300f43511e-kube-api-access-sg7cz" pod "79f228f5-1947-426c-9e4d-4f300f43511e" (UID: "79f228f5-1947-426c-9e4d-4f300f43511e"). InnerVolumeSpecName "kube-api-access-sg7cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:01:56.364156 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.364133    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f228f5-1947-426c-9e4d-4f300f43511e-istio-token" pod "79f228f5-1947-426c-9e4d-4f300f43511e" (UID: "79f228f5-1947-426c-9e4d-4f300f43511e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:01:56.463518 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.463440    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79f228f5-1947-426c-9e4d-4f300f43511e-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:56.463518 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.463468    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79f228f5-1947-426c-9e4d-4f300f43511e-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:56.463518 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.463481    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg7cz\" (UniqueName: \"kubernetes.io/projected/79f228f5-1947-426c-9e4d-4f300f43511e-kube-api-access-sg7cz\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:56.463518 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.463491    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79f228f5-1947-426c-9e4d-4f300f43511e-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:56.565421 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.565393    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 01:01:56.666744 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.666706    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-kube-api-access-kttf5\" (UniqueName: \"kubernetes.io/projected/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-kube-api-access-kttf5\") pod \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") "
Oct 31 01:01:56.666933 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.666819    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-podinfo\") pod \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") "
Oct 31 01:01:56.666996 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.666941    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-envoy\") pod \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") "
Oct 31 01:01:56.666996 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.666976    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-credential-socket\") pod \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") "
Oct 31 01:01:56.667131 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.667049    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-workload-certs\") pod \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") "
Oct 31 01:01:56.667131 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.667079    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-workload-socket\") pod \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") "
Oct 31 01:01:56.667235 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.667136    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istiod-ca-cert\") pod \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") "
Oct 31 01:01:56.667291 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.667245    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-data\") pod \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") "
Oct 31 01:01:56.667355 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.667334    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-token\" (UniqueName: \"kubernetes.io/projected/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-token\") pod \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\" (UID: \"47727a40-911f-4d8d-8fd6-f19ada8e4ea0\") "
Oct 31 01:01:56.667471 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.667439    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-workload-certs" pod "47727a40-911f-4d8d-8fd6-f19ada8e4ea0" (UID: "47727a40-911f-4d8d-8fd6-f19ada8e4ea0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:56.667842 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.667788    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-credential-socket" pod "47727a40-911f-4d8d-8fd6-f19ada8e4ea0" (UID: "47727a40-911f-4d8d-8fd6-f19ada8e4ea0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:56.667842 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.667802    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-workload-socket" pod "47727a40-911f-4d8d-8fd6-f19ada8e4ea0" (UID: "47727a40-911f-4d8d-8fd6-f19ada8e4ea0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:56.667842 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.667787    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-data" pod "47727a40-911f-4d8d-8fd6-f19ada8e4ea0" (UID: "47727a40-911f-4d8d-8fd6-f19ada8e4ea0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:56.668077 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.668044    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:56.668077 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.668068    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:56.668192 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.668082    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:56.668192 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.668096    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:56.668659 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.668631    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istiod-ca-cert" pod "47727a40-911f-4d8d-8fd6-f19ada8e4ea0" (UID: "47727a40-911f-4d8d-8fd6-f19ada8e4ea0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:01:56.669726 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.669697    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-envoy" pod "47727a40-911f-4d8d-8fd6-f19ada8e4ea0" (UID: "47727a40-911f-4d8d-8fd6-f19ada8e4ea0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:56.669833 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.669817    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-podinfo" pod "47727a40-911f-4d8d-8fd6-f19ada8e4ea0" (UID: "47727a40-911f-4d8d-8fd6-f19ada8e4ea0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:01:56.670260 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.670233    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-token" pod "47727a40-911f-4d8d-8fd6-f19ada8e4ea0" (UID: "47727a40-911f-4d8d-8fd6-f19ada8e4ea0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:01:56.671152 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.671125    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-kube-api-access-kttf5" pod "47727a40-911f-4d8d-8fd6-f19ada8e4ea0" (UID: "47727a40-911f-4d8d-8fd6-f19ada8e4ea0"). InnerVolumeSpecName "kube-api-access-kttf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:01:56.769617 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.769514    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:56.769617 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.769551    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:56.769617 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.769564    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kttf5\" (UniqueName: \"kubernetes.io/projected/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-kube-api-access-kttf5\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:56.769617 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.769578    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:56.769617 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:56.769590    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47727a40-911f-4d8d-8fd6-f19ada8e4ea0-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:01:57.118288 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:57.118247    4327 generic.go:356] "Generic (PLEG): container finished" podID="47727a40-911f-4d8d-8fd6-f19ada8e4ea0" containerID="b94f41e5ac21207e05812b0614472387e51061e4bd6c94b34a6a2e58c681779f" exitCode=0
Oct 31 01:01:57.118772 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:57.118299    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"
Oct 31 01:01:57.118772 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:57.118334    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv" event={"ID":"47727a40-911f-4d8d-8fd6-f19ada8e4ea0","Type":"ContainerDied","Data":"b94f41e5ac21207e05812b0614472387e51061e4bd6c94b34a6a2e58c681779f"}
Oct 31 01:01:57.118772 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:57.118386    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv" event={"ID":"47727a40-911f-4d8d-8fd6-f19ada8e4ea0","Type":"ContainerDied","Data":"72a457f6907b123accf13fe1477d365c8dccf615cd4dd40dfe058ecd6e1e59cd"}
Oct 31 01:01:57.118772 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:57.118410    4327 scope.go:122] "RemoveContainer" containerID="b94f41e5ac21207e05812b0614472387e51061e4bd6c94b34a6a2e58c681779f"
Oct 31 01:01:57.120125 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:57.120101    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"
Oct 31 01:01:57.120266 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:57.120105    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d" event={"ID":"79f228f5-1947-426c-9e4d-4f300f43511e","Type":"ContainerDied","Data":"661ef34188433bb35935d46a8dde691777eed1983e62b43b2bcde074408c86d6"}
Oct 31 01:01:57.133004 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:57.132981    4327 scope.go:122] "RemoveContainer" containerID="8514da30c40f9a548d39bd46eb762a911457d8b78e6f4b792b4a77dfa98caa47"
Oct 31 01:01:57.144824 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:57.144802    4327 scope.go:122] "RemoveContainer" containerID="b94f41e5ac21207e05812b0614472387e51061e4bd6c94b34a6a2e58c681779f"
Oct 31 01:01:57.145126 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:01:57.145096    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b94f41e5ac21207e05812b0614472387e51061e4bd6c94b34a6a2e58c681779f\": container with ID starting with b94f41e5ac21207e05812b0614472387e51061e4bd6c94b34a6a2e58c681779f not found: ID does not exist" containerID="b94f41e5ac21207e05812b0614472387e51061e4bd6c94b34a6a2e58c681779f"
Oct 31 01:01:57.145236 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:57.145136    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b94f41e5ac21207e05812b0614472387e51061e4bd6c94b34a6a2e58c681779f"} err="failed to get container status \"b94f41e5ac21207e05812b0614472387e51061e4bd6c94b34a6a2e58c681779f\": rpc error: code = NotFound desc = could not find container \"b94f41e5ac21207e05812b0614472387e51061e4bd6c94b34a6a2e58c681779f\": container with ID starting with b94f41e5ac21207e05812b0614472387e51061e4bd6c94b34a6a2e58c681779f not found: ID does not exist"
Oct 31 01:01:57.145236 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:57.145161    4327 scope.go:122] "RemoveContainer" containerID="8514da30c40f9a548d39bd46eb762a911457d8b78e6f4b792b4a77dfa98caa47"
Oct 31 01:01:57.145525 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:01:57.145497    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8514da30c40f9a548d39bd46eb762a911457d8b78e6f4b792b4a77dfa98caa47\": container with ID starting with 8514da30c40f9a548d39bd46eb762a911457d8b78e6f4b792b4a77dfa98caa47 not found: ID does not exist" containerID="8514da30c40f9a548d39bd46eb762a911457d8b78e6f4b792b4a77dfa98caa47"
Oct 31 01:01:57.145582 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:57.145529    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8514da30c40f9a548d39bd46eb762a911457d8b78e6f4b792b4a77dfa98caa47"} err="failed to get container status \"8514da30c40f9a548d39bd46eb762a911457d8b78e6f4b792b4a77dfa98caa47\": rpc error: code = NotFound desc = could not find container \"8514da30c40f9a548d39bd46eb762a911457d8b78e6f4b792b4a77dfa98caa47\": container with ID starting with 8514da30c40f9a548d39bd46eb762a911457d8b78e6f4b792b4a77dfa98caa47 not found: ID does not exist"
Oct 31 01:01:57.145582 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:57.145553    4327 scope.go:122] "RemoveContainer" containerID="1c7268daabd20de5cb1fc2b74dab2150c455e7a3a1d83323d5c65bc2b519388e"
Oct 31 01:01:57.179211 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:57.179187    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"]
Oct 31 01:01:57.183461 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:57.183437    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fd1dfc-openshift-default-75bbcc65dc-jcvjv"]
Oct 31 01:01:57.227748 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:57.227715    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"]
Oct 31 01:01:57.234007 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:57.233978    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c6d97d-openshift-default-79fc67f7c9-k5r4d"]
Oct 31 01:01:57.925772 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:57.925738    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47727a40-911f-4d8d-8fd6-f19ada8e4ea0" path="/var/lib/kubelet/pods/47727a40-911f-4d8d-8fd6-f19ada8e4ea0/volumes"
Oct 31 01:01:57.926415 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:57.926392    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79f228f5-1947-426c-9e4d-4f300f43511e" path="/var/lib/kubelet/pods/79f228f5-1947-426c-9e4d-4f300f43511e/volumes"
Oct 31 01:01:58.850824 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:58.850781    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl"]
Oct 31 01:01:58.851975 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:58.851944    4327 cpu_manager.go:414] "containerMap: removing container" podUID="79f228f5-1947-426c-9e4d-4f300f43511e" containerName="istio-proxy"
Oct 31 01:01:58.851975 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:58.851968    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f228f5-1947-426c-9e4d-4f300f43511e" containerName="istio-proxy"
Oct 31 01:01:58.852172 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:58.851996    4327 cpu_manager.go:414] "containerMap: removing container" podUID="47727a40-911f-4d8d-8fd6-f19ada8e4ea0" containerName="istio-proxy"
Oct 31 01:01:58.852172 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:58.852003    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47727a40-911f-4d8d-8fd6-f19ada8e4ea0" containerName="istio-proxy"
Oct 31 01:01:58.852172 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:58.852026    4327 cpu_manager.go:414] "containerMap: removing container" podUID="79f228f5-1947-426c-9e4d-4f300f43511e" containerName="istio-proxy"
Oct 31 01:01:58.852172 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:58.852035    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f228f5-1947-426c-9e4d-4f300f43511e" containerName="istio-proxy"
Oct 31 01:01:58.852376 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:58.852244    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="79f228f5-1947-426c-9e4d-4f300f43511e" containerName="istio-proxy"
Oct 31 01:01:58.852376 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:58.852261    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="47727a40-911f-4d8d-8fd6-f19ada8e4ea0" containerName="istio-proxy"
Oct 31 01:01:58.852376 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:58.852272    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="79f228f5-1947-426c-9e4d-4f300f43511e" containerName="istio-proxy"
Oct 31 01:01:58.852376 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:58.852289    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="47727a40-911f-4d8d-8fd6-f19ada8e4ea0" containerName="istio-proxy"
Oct 31 01:01:58.857031 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:58.857007    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj"]
Oct 31 01:01:58.857157 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:58.857119    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl"
Oct 31 01:01:58.858060 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:58.858037    4327 cpu_manager.go:414] "containerMap: removing container" podUID="47727a40-911f-4d8d-8fd6-f19ada8e4ea0" containerName="istio-proxy"
Oct 31 01:01:58.858060 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:58.858059    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47727a40-911f-4d8d-8fd6-f19ada8e4ea0" containerName="istio-proxy"
Oct 31 01:01:58.861734 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:58.861717    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj"
Oct 31 01:01:58.881656 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:58.881632    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl"]
Oct 31 01:01:58.884575 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:58.884551    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj"]
Oct 31 01:01:58.887118 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:58.887095    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-476a67\""
Oct 31 01:01:58.887265 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:58.887240    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-476a67-dockercfg-kvk7j\""
Oct 31 01:01:58.899277 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:58.899246    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl\" (UID: \"ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl"
Oct 31 01:01:58.899466 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:58.899434    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnt2b\" (UniqueName: \"kubernetes.io/projected/ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0-kube-api-access-wnt2b\") pod \"payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl\" (UID: \"ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl"
Oct 31 01:01:58.899543 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:58.899492    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5575\" (UniqueName: \"kubernetes.io/projected/f15fde05-d821-4a9f-bff6-d2ce68887b8b-kube-api-access-r5575\") pod \"payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj\" (UID: \"f15fde05-d821-4a9f-bff6-d2ce68887b8b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj"
Oct 31 01:01:58.899671 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:58.899621    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f15fde05-d821-4a9f-bff6-d2ce68887b8b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj\" (UID: \"f15fde05-d821-4a9f-bff6-d2ce68887b8b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj"
Oct 31 01:01:59.019701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:59.003612    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl\" (UID: \"ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl"
Oct 31 01:01:59.019701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:59.003735    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnt2b\" (UniqueName: \"kubernetes.io/projected/ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0-kube-api-access-wnt2b\") pod \"payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl\" (UID: \"ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl"
Oct 31 01:01:59.019701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:59.003783    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5575\" (UniqueName: \"kubernetes.io/projected/f15fde05-d821-4a9f-bff6-d2ce68887b8b-kube-api-access-r5575\") pod \"payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj\" (UID: \"f15fde05-d821-4a9f-bff6-d2ce68887b8b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj"
Oct 31 01:01:59.019701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:59.003879    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f15fde05-d821-4a9f-bff6-d2ce68887b8b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj\" (UID: \"f15fde05-d821-4a9f-bff6-d2ce68887b8b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj"
Oct 31 01:01:59.045202 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:59.045168    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl\" (UID: \"ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl"
Oct 31 01:01:59.061698 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:59.060192    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f15fde05-d821-4a9f-bff6-d2ce68887b8b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj\" (UID: \"f15fde05-d821-4a9f-bff6-d2ce68887b8b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj"
Oct 31 01:01:59.093769 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:59.090717    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnt2b\" (UniqueName: \"kubernetes.io/projected/ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0-kube-api-access-wnt2b\") pod \"payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl\" (UID: \"ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl"
Oct 31 01:01:59.119571 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:59.117442    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5575\" (UniqueName: \"kubernetes.io/projected/f15fde05-d821-4a9f-bff6-d2ce68887b8b-kube-api-access-r5575\") pod \"payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj\" (UID: \"f15fde05-d821-4a9f-bff6-d2ce68887b8b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj"
Oct 31 01:01:59.177217 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:59.173874    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl"
Oct 31 01:01:59.182961 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:59.182925    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj"
Oct 31 01:01:59.497273 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:59.497244    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl"]
Oct 31 01:01:59.497400 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:01:59.497252    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc9ebd5_e3d6_48ed_9a8e_c11e7ef200e0.slice/crio-a07b0fe44a9fdb25fcc4570bf8edd524e957b0d807d9a20df9dce1520bb8012b WatchSource:0}: Error finding container a07b0fe44a9fdb25fcc4570bf8edd524e957b0d807d9a20df9dce1520bb8012b: Status 404 returned error can't find the container with id a07b0fe44a9fdb25fcc4570bf8edd524e957b0d807d9a20df9dce1520bb8012b
Oct 31 01:01:59.519283 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:01:59.519250    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf15fde05_d821_4a9f_bff6_d2ce68887b8b.slice/crio-1731aafc59211dcbe230e79ed69c8091f117b83a7b9b4376b785eea0320208f5 WatchSource:0}: Error finding container 1731aafc59211dcbe230e79ed69c8091f117b83a7b9b4376b785eea0320208f5: Status 404 returned error can't find the container with id 1731aafc59211dcbe230e79ed69c8091f117b83a7b9b4376b785eea0320208f5
Oct 31 01:01:59.521368 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:01:59.521342    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj"]
Oct 31 01:02:00.164002 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:00.163961    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj" event={"ID":"f15fde05-d821-4a9f-bff6-d2ce68887b8b","Type":"ContainerStarted","Data":"122d6c5a980a807f0da62a630b6d86c1124a3f07590131950e8b11fc996b1891"}
Oct 31 01:02:00.164474 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:00.164030    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj" event={"ID":"f15fde05-d821-4a9f-bff6-d2ce68887b8b","Type":"ContainerStarted","Data":"1731aafc59211dcbe230e79ed69c8091f117b83a7b9b4376b785eea0320208f5"}
Oct 31 01:02:00.164474 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:00.164183    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj"
Oct 31 01:02:00.165580 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:00.165557    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl" event={"ID":"ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0","Type":"ContainerStarted","Data":"e50d732ba35f2960aa6409d0f8997b9e77528e93d72acd23e25c9d289d982add"}
Oct 31 01:02:00.165726 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:00.165597    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl" event={"ID":"ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0","Type":"ContainerStarted","Data":"a07b0fe44a9fdb25fcc4570bf8edd524e957b0d807d9a20df9dce1520bb8012b"}
Oct 31 01:02:00.165726 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:00.165631    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl"
Oct 31 01:02:00.209911 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:00.209858    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj" podStartSLOduration=2.209844994 podStartE2EDuration="2.209844994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:02:00.207466233 +0000 UTC m=+2580.429501775" watchObservedRunningTime="2026-10-31 01:02:00.209844994 +0000 UTC m=+2580.431880538"
Oct 31 01:02:00.258604 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:00.258554    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl" podStartSLOduration=2.258539438 podStartE2EDuration="2.258539438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:02:00.256206914 +0000 UTC m=+2580.478242456" watchObservedRunningTime="2026-10-31 01:02:00.258539438 +0000 UTC m=+2580.480575029"
Oct 31 01:02:02.441813 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:02.441774    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-609843-b8d75c975-frvqn"]
Oct 31 01:02:02.480509 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:02.480470    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t"]
Oct 31 01:02:07.442732 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:07.442638    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-609843-b8d75c975-frvqn" podUID="80b7e0a0-98d7-4176-9657-dcd37514545b" containerName="payload-processing" containerID="cri-o://f9f4eb5ebfc640fb770e1006ae2ebad7bbf53a96e779f380e831954e4104c986" gracePeriod=25
Oct 31 01:02:07.481090 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:07.481044    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t" podUID="2108960e-cdf0-40c8-a646-dbdcecc96099" containerName="payload-pre-processing" containerID="cri-o://97ac57aa39f8bb899bb03c06c26065c94b54f3d2ebbcad52a2dc30f3e2327ace" gracePeriod=25
Oct 31 01:02:07.737275 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:07.737244    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-609843-b8d75c975-frvqn"
Oct 31 01:02:07.881376 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:07.881340    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80b7e0a0-98d7-4176-9657-dcd37514545b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b7e0a0-98d7-4176-9657-dcd37514545b-plugins-config-volume\") pod \"80b7e0a0-98d7-4176-9657-dcd37514545b\" (UID: \"80b7e0a0-98d7-4176-9657-dcd37514545b\") "
Oct 31 01:02:07.881556 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:07.881478    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b7e0a0-98d7-4176-9657-dcd37514545b-kube-api-access-whm45\" (UniqueName: \"kubernetes.io/projected/80b7e0a0-98d7-4176-9657-dcd37514545b-kube-api-access-whm45\") pod \"80b7e0a0-98d7-4176-9657-dcd37514545b\" (UID: \"80b7e0a0-98d7-4176-9657-dcd37514545b\") "
Oct 31 01:02:07.881782 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:07.881760    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b7e0a0-98d7-4176-9657-dcd37514545b-plugins-config-volume" pod "80b7e0a0-98d7-4176-9657-dcd37514545b" (UID: "80b7e0a0-98d7-4176-9657-dcd37514545b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:02:07.882066 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:07.882050    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b7e0a0-98d7-4176-9657-dcd37514545b-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:02:07.883692 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:07.883656    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b7e0a0-98d7-4176-9657-dcd37514545b-kube-api-access-whm45" pod "80b7e0a0-98d7-4176-9657-dcd37514545b" (UID: "80b7e0a0-98d7-4176-9657-dcd37514545b"). InnerVolumeSpecName "kube-api-access-whm45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:02:07.983736 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:07.983618    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whm45\" (UniqueName: \"kubernetes.io/projected/80b7e0a0-98d7-4176-9657-dcd37514545b-kube-api-access-whm45\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:02:08.259364 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:08.259266    4327 generic.go:356] "Generic (PLEG): container finished" podID="80b7e0a0-98d7-4176-9657-dcd37514545b" containerID="f9f4eb5ebfc640fb770e1006ae2ebad7bbf53a96e779f380e831954e4104c986" exitCode=0
Oct 31 01:02:08.259364 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:08.259321    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-609843-b8d75c975-frvqn"
Oct 31 01:02:08.259594 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:08.259357    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-609843-b8d75c975-frvqn" event={"ID":"80b7e0a0-98d7-4176-9657-dcd37514545b","Type":"ContainerDied","Data":"f9f4eb5ebfc640fb770e1006ae2ebad7bbf53a96e779f380e831954e4104c986"}
Oct 31 01:02:08.259594 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:08.259413    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-609843-b8d75c975-frvqn" event={"ID":"80b7e0a0-98d7-4176-9657-dcd37514545b","Type":"ContainerDied","Data":"2d2c3cb23b0bedd9246d57b2ee95c5afd945a16d49120a41d275878bba717354"}
Oct 31 01:02:08.259594 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:08.259436    4327 scope.go:122] "RemoveContainer" containerID="f9f4eb5ebfc640fb770e1006ae2ebad7bbf53a96e779f380e831954e4104c986"
Oct 31 01:02:08.270466 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:08.270436    4327 scope.go:122] "RemoveContainer" containerID="f9f4eb5ebfc640fb770e1006ae2ebad7bbf53a96e779f380e831954e4104c986"
Oct 31 01:02:08.270868 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:02:08.270843    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9f4eb5ebfc640fb770e1006ae2ebad7bbf53a96e779f380e831954e4104c986\": container with ID starting with f9f4eb5ebfc640fb770e1006ae2ebad7bbf53a96e779f380e831954e4104c986 not found: ID does not exist" containerID="f9f4eb5ebfc640fb770e1006ae2ebad7bbf53a96e779f380e831954e4104c986"
Oct 31 01:02:08.270995 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:08.270876    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f4eb5ebfc640fb770e1006ae2ebad7bbf53a96e779f380e831954e4104c986"} err="failed to get container status \"f9f4eb5ebfc640fb770e1006ae2ebad7bbf53a96e779f380e831954e4104c986\": rpc error: code = NotFound desc = could not find container \"f9f4eb5ebfc640fb770e1006ae2ebad7bbf53a96e779f380e831954e4104c986\": container with ID starting with f9f4eb5ebfc640fb770e1006ae2ebad7bbf53a96e779f380e831954e4104c986 not found: ID does not exist"
Oct 31 01:02:08.323255 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:08.323216    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-609843-b8d75c975-frvqn"]
Oct 31 01:02:08.337634 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:08.337598    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-609843-b8d75c975-frvqn"]
Oct 31 01:02:08.661112 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:08.661078    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t" podUID="2108960e-cdf0-40c8-a646-dbdcecc96099" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:02:08.677198 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:08.677159    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-609843-b8d75c975-frvqn" podUID="80b7e0a0-98d7-4176-9657-dcd37514545b" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.142:50052\" within 1s: context deadline exceeded"
Oct 31 01:02:08.677377 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:02:08.677233    4327 logging.go:55] [core] [Channel #6806 SubChannel #6807]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.142:50052", ServerName: "10.133.0.142:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.142:50052: operation was canceled"
Oct 31 01:02:09.923349 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:09.923314    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80b7e0a0-98d7-4176-9657-dcd37514545b" path="/var/lib/kubelet/pods/80b7e0a0-98d7-4176-9657-dcd37514545b/volumes"
Oct 31 01:02:11.178851 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:11.178817    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj"
Oct 31 01:02:11.179271 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:11.179186    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl"
Oct 31 01:02:15.294213 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:02:15.294158    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:02:18.660950 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:18.660893    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t" podUID="2108960e-cdf0-40c8-a646-dbdcecc96099" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:02:27.772837 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:27.772808    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t"
Oct 31 01:02:27.947298 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:27.947266    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2108960e-cdf0-40c8-a646-dbdcecc96099-kube-api-access-bhtpv\" (UniqueName: \"kubernetes.io/projected/2108960e-cdf0-40c8-a646-dbdcecc96099-kube-api-access-bhtpv\") pod \"2108960e-cdf0-40c8-a646-dbdcecc96099\" (UID: \"2108960e-cdf0-40c8-a646-dbdcecc96099\") "
Oct 31 01:02:27.947489 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:27.947367    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2108960e-cdf0-40c8-a646-dbdcecc96099-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2108960e-cdf0-40c8-a646-dbdcecc96099-plugins-config-volume\") pod \"2108960e-cdf0-40c8-a646-dbdcecc96099\" (UID: \"2108960e-cdf0-40c8-a646-dbdcecc96099\") "
Oct 31 01:02:27.947776 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:27.947752    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2108960e-cdf0-40c8-a646-dbdcecc96099-plugins-config-volume" pod "2108960e-cdf0-40c8-a646-dbdcecc96099" (UID: "2108960e-cdf0-40c8-a646-dbdcecc96099"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:02:27.948050 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:27.948034    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2108960e-cdf0-40c8-a646-dbdcecc96099-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:02:27.949498 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:27.949475    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2108960e-cdf0-40c8-a646-dbdcecc96099-kube-api-access-bhtpv" pod "2108960e-cdf0-40c8-a646-dbdcecc96099" (UID: "2108960e-cdf0-40c8-a646-dbdcecc96099"). InnerVolumeSpecName "kube-api-access-bhtpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:02:28.049855 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:28.049772    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhtpv\" (UniqueName: \"kubernetes.io/projected/2108960e-cdf0-40c8-a646-dbdcecc96099-kube-api-access-bhtpv\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:02:28.459365 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:28.459327    4327 generic.go:356] "Generic (PLEG): container finished" podID="2108960e-cdf0-40c8-a646-dbdcecc96099" containerID="97ac57aa39f8bb899bb03c06c26065c94b54f3d2ebbcad52a2dc30f3e2327ace" exitCode=0
Oct 31 01:02:28.459577 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:28.459374    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t"
Oct 31 01:02:28.459577 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:28.459412    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t" event={"ID":"2108960e-cdf0-40c8-a646-dbdcecc96099","Type":"ContainerDied","Data":"97ac57aa39f8bb899bb03c06c26065c94b54f3d2ebbcad52a2dc30f3e2327ace"}
Oct 31 01:02:28.459577 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:28.459477    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t" event={"ID":"2108960e-cdf0-40c8-a646-dbdcecc96099","Type":"ContainerDied","Data":"35465f255f504aa93824a44b5cddb82cb86ff2ee3f2faf80f61be0f29070f389"}
Oct 31 01:02:28.459577 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:28.459507    4327 scope.go:122] "RemoveContainer" containerID="97ac57aa39f8bb899bb03c06c26065c94b54f3d2ebbcad52a2dc30f3e2327ace"
Oct 31 01:02:28.471260 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:28.471235    4327 scope.go:122] "RemoveContainer" containerID="97ac57aa39f8bb899bb03c06c26065c94b54f3d2ebbcad52a2dc30f3e2327ace"
Oct 31 01:02:28.471567 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:02:28.471545    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97ac57aa39f8bb899bb03c06c26065c94b54f3d2ebbcad52a2dc30f3e2327ace\": container with ID starting with 97ac57aa39f8bb899bb03c06c26065c94b54f3d2ebbcad52a2dc30f3e2327ace not found: ID does not exist" containerID="97ac57aa39f8bb899bb03c06c26065c94b54f3d2ebbcad52a2dc30f3e2327ace"
Oct 31 01:02:28.471626 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:28.471576    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97ac57aa39f8bb899bb03c06c26065c94b54f3d2ebbcad52a2dc30f3e2327ace"} err="failed to get container status \"97ac57aa39f8bb899bb03c06c26065c94b54f3d2ebbcad52a2dc30f3e2327ace\": rpc error: code = NotFound desc = could not find container \"97ac57aa39f8bb899bb03c06c26065c94b54f3d2ebbcad52a2dc30f3e2327ace\": container with ID starting with 97ac57aa39f8bb899bb03c06c26065c94b54f3d2ebbcad52a2dc30f3e2327ace not found: ID does not exist"
Oct 31 01:02:28.518600 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:28.518570    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t"]
Oct 31 01:02:28.530403 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:28.530367    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-609843-79599884c-vjw2t"]
Oct 31 01:02:29.923490 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:29.923459    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2108960e-cdf0-40c8-a646-dbdcecc96099" path="/var/lib/kubelet/pods/2108960e-cdf0-40c8-a646-dbdcecc96099/volumes"
Oct 31 01:02:38.551612 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:38.551573    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq"]
Oct 31 01:02:38.552381 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:38.552365    4327 cpu_manager.go:414] "containerMap: removing container" podUID="80b7e0a0-98d7-4176-9657-dcd37514545b" containerName="payload-processing"
Oct 31 01:02:38.552443 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:38.552382    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b7e0a0-98d7-4176-9657-dcd37514545b" containerName="payload-processing"
Oct 31 01:02:38.552443 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:38.552402    4327 cpu_manager.go:414] "containerMap: removing container" podUID="2108960e-cdf0-40c8-a646-dbdcecc96099" containerName="payload-pre-processing"
Oct 31 01:02:38.552443 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:38.552409    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2108960e-cdf0-40c8-a646-dbdcecc96099" containerName="payload-pre-processing"
Oct 31 01:02:38.552559 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:38.552532    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b7e0a0-98d7-4176-9657-dcd37514545b" containerName="payload-processing"
Oct 31 01:02:38.552559 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:38.552543    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="2108960e-cdf0-40c8-a646-dbdcecc96099" containerName="payload-pre-processing"
Oct 31 01:02:38.557971 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:38.557945    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq"
Oct 31 01:02:38.578694 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:38.578630    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-476a67-serving-cert\""
Oct 31 01:02:38.579637 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:38.579606    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq"]
Oct 31 01:02:38.729948 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:38.729904    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq\" (UID: \"273665c2-c5c0-4a57-930e-cfbd27ec4fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq"
Oct 31 01:02:38.730168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:38.729980    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq\" (UID: \"273665c2-c5c0-4a57-930e-cfbd27ec4fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq"
Oct 31 01:02:38.730168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:38.730149    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpdx6\" (UniqueName: \"kubernetes.io/projected/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-kube-api-access-qpdx6\") pod \"maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq\" (UID: \"273665c2-c5c0-4a57-930e-cfbd27ec4fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq"
Oct 31 01:02:38.730354 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:38.730329    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq\" (UID: \"273665c2-c5c0-4a57-930e-cfbd27ec4fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq"
Oct 31 01:02:38.832583 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:38.832482    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq\" (UID: \"273665c2-c5c0-4a57-930e-cfbd27ec4fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq"
Oct 31 01:02:38.832974 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:38.832951    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq\" (UID: \"273665c2-c5c0-4a57-930e-cfbd27ec4fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq"
Oct 31 01:02:38.833187 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:38.833167    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq\" (UID: \"273665c2-c5c0-4a57-930e-cfbd27ec4fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq"
Oct 31 01:02:38.833792 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:38.833769    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpdx6\" (UniqueName: \"kubernetes.io/projected/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-kube-api-access-qpdx6\") pod \"maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq\" (UID: \"273665c2-c5c0-4a57-930e-cfbd27ec4fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq"
Oct 31 01:02:38.834479 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:38.834353    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq\" (UID: \"273665c2-c5c0-4a57-930e-cfbd27ec4fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq"
Oct 31 01:02:38.840013 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:38.839988    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq\" (UID: \"273665c2-c5c0-4a57-930e-cfbd27ec4fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq"
Oct 31 01:02:38.842662 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:38.842636    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq\" (UID: \"273665c2-c5c0-4a57-930e-cfbd27ec4fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq"
Oct 31 01:02:38.858257 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:38.858224    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpdx6\" (UniqueName: \"kubernetes.io/projected/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-kube-api-access-qpdx6\") pod \"maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq\" (UID: \"273665c2-c5c0-4a57-930e-cfbd27ec4fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq"
Oct 31 01:02:38.876258 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:38.876233    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq"
Oct 31 01:02:39.188790 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:39.188757    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq"]
Oct 31 01:02:39.189834 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:02:39.189805    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod273665c2_c5c0_4a57_930e_cfbd27ec4fe5.slice/crio-6fca16f2d36c4624f4939263cc2123a4e3acdf4b086cc113eba3f2a1b84922d7 WatchSource:0}: Error finding container 6fca16f2d36c4624f4939263cc2123a4e3acdf4b086cc113eba3f2a1b84922d7: Status 404 returned error can't find the container with id 6fca16f2d36c4624f4939263cc2123a4e3acdf4b086cc113eba3f2a1b84922d7
Oct 31 01:02:39.192098 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:39.192077    4327 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:02:39.586462 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:39.586417    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq" event={"ID":"273665c2-c5c0-4a57-930e-cfbd27ec4fe5","Type":"ContainerStarted","Data":"6fca16f2d36c4624f4939263cc2123a4e3acdf4b086cc113eba3f2a1b84922d7"}
Oct 31 01:02:40.596828 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:40.596788    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq" event={"ID":"273665c2-c5c0-4a57-930e-cfbd27ec4fe5","Type":"ContainerStarted","Data":"7073f13efcf956bdcc5148c94e28f37440b0e711bf1c50b34309cb9d81a6ac51"}
Oct 31 01:02:40.597347 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:40.596899    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq"
Oct 31 01:02:45.295180 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:02:45.295136    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:02:46.611499 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:46.611469    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq"
Oct 31 01:02:46.659182 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:46.659119    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq" podStartSLOduration=8.270351197 podStartE2EDuration="8.659104385s" totalImagesPullingTime="388.753188ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:02:40.690137837 +0000 UTC m=+2620.912173379" watchObservedRunningTime="2026-10-31 01:02:46.659104385 +0000 UTC m=+2626.881139927"
Oct 31 01:02:52.961628 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:52.961594    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl"]
Oct 31 01:02:52.966712 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:52.966693    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl"
Oct 31 01:02:52.986582 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:52.986551    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl"]
Oct 31 01:02:52.995958 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:52.995916    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-99a5c2\""
Oct 31 01:02:52.996294 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:52.996276    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-99a5c2-dockercfg-ql2sj\""
Oct 31 01:02:53.061705 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:53.054564    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg5wv\" (UniqueName: \"kubernetes.io/projected/9dcfe474-6b38-4ada-9272-a48f011722b3-kube-api-access-wg5wv\") pod \"payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl\" (UID: \"9dcfe474-6b38-4ada-9272-a48f011722b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl"
Oct 31 01:02:53.061705 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:53.054917    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dcfe474-6b38-4ada-9272-a48f011722b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl\" (UID: \"9dcfe474-6b38-4ada-9272-a48f011722b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl"
Oct 31 01:02:53.168626 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:53.168581    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dcfe474-6b38-4ada-9272-a48f011722b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl\" (UID: \"9dcfe474-6b38-4ada-9272-a48f011722b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl"
Oct 31 01:02:53.177626 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:53.177572    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg5wv\" (UniqueName: \"kubernetes.io/projected/9dcfe474-6b38-4ada-9272-a48f011722b3-kube-api-access-wg5wv\") pod \"payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl\" (UID: \"9dcfe474-6b38-4ada-9272-a48f011722b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl"
Oct 31 01:02:53.178376 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:53.174897    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dcfe474-6b38-4ada-9272-a48f011722b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl\" (UID: \"9dcfe474-6b38-4ada-9272-a48f011722b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl"
Oct 31 01:02:53.193327 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:53.191958    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr"]
Oct 31 01:02:53.199903 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:53.199401    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr"
Oct 31 01:02:53.221362 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:53.221328    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr"]
Oct 31 01:02:53.229228 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:53.228134    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg5wv\" (UniqueName: \"kubernetes.io/projected/9dcfe474-6b38-4ada-9272-a48f011722b3-kube-api-access-wg5wv\") pod \"payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl\" (UID: \"9dcfe474-6b38-4ada-9272-a48f011722b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl"
Oct 31 01:02:53.279733 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:53.279634    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt2xd\" (UniqueName: \"kubernetes.io/projected/c853f186-bade-4124-b526-438799578a2c-kube-api-access-kt2xd\") pod \"payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr\" (UID: \"c853f186-bade-4124-b526-438799578a2c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr"
Oct 31 01:02:53.280071 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:53.280017    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c853f186-bade-4124-b526-438799578a2c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr\" (UID: \"c853f186-bade-4124-b526-438799578a2c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr"
Oct 31 01:02:53.288552 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:53.288526    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl"
Oct 31 01:02:53.382073 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:53.382038    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt2xd\" (UniqueName: \"kubernetes.io/projected/c853f186-bade-4124-b526-438799578a2c-kube-api-access-kt2xd\") pod \"payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr\" (UID: \"c853f186-bade-4124-b526-438799578a2c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr"
Oct 31 01:02:53.382246 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:53.382137    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c853f186-bade-4124-b526-438799578a2c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr\" (UID: \"c853f186-bade-4124-b526-438799578a2c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr"
Oct 31 01:02:53.382856 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:53.382836    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c853f186-bade-4124-b526-438799578a2c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr\" (UID: \"c853f186-bade-4124-b526-438799578a2c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr"
Oct 31 01:02:53.408510 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:53.408478    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt2xd\" (UniqueName: \"kubernetes.io/projected/c853f186-bade-4124-b526-438799578a2c-kube-api-access-kt2xd\") pod \"payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr\" (UID: \"c853f186-bade-4124-b526-438799578a2c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr"
Oct 31 01:02:53.531335 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:53.531242    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr"
Oct 31 01:02:53.602392 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:53.602360    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl"]
Oct 31 01:02:53.803728 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:53.803621    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl" event={"ID":"9dcfe474-6b38-4ada-9272-a48f011722b3","Type":"ContainerStarted","Data":"c67c8fa82494def93542d40cbe743e1306a074e8370d63320f3ece0a5178bfac"}
Oct 31 01:02:53.803728 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:53.803722    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl" event={"ID":"9dcfe474-6b38-4ada-9272-a48f011722b3","Type":"ContainerStarted","Data":"2eeaa57605f99059c9b46695744e7ea2e93620861c508aa49a55a57c5824162d"}
Oct 31 01:02:53.803974 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:53.803856    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl"
Oct 31 01:02:53.862179 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:02:53.862150    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc853f186_bade_4124_b526_438799578a2c.slice/crio-806b4916e74ae44917e6f3481938ae00ee58e31760a2a0506b594447a58f5dc4 WatchSource:0}: Error finding container 806b4916e74ae44917e6f3481938ae00ee58e31760a2a0506b594447a58f5dc4: Status 404 returned error can't find the container with id 806b4916e74ae44917e6f3481938ae00ee58e31760a2a0506b594447a58f5dc4
Oct 31 01:02:53.864391 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:53.864370    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr"]
Oct 31 01:02:53.886808 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:53.886757    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl" podStartSLOduration=1.886741572 podStartE2EDuration="1.886741572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:02:53.884414483 +0000 UTC m=+2634.106450026" watchObservedRunningTime="2026-10-31 01:02:53.886741572 +0000 UTC m=+2634.108777114"
Oct 31 01:02:54.823470 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:54.823417    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr" event={"ID":"c853f186-bade-4124-b526-438799578a2c","Type":"ContainerStarted","Data":"042d70cc4386f58baf06a5b6222cc988dc62e9693ed0a984a19734153f786939"}
Oct 31 01:02:54.823941 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:54.823530    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr"
Oct 31 01:02:54.823941 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:54.823552    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr" event={"ID":"c853f186-bade-4124-b526-438799578a2c","Type":"ContainerStarted","Data":"806b4916e74ae44917e6f3481938ae00ee58e31760a2a0506b594447a58f5dc4"}
Oct 31 01:02:54.865922 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:02:54.865867    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr" podStartSLOduration=1.865854817 podStartE2EDuration="1.865854817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:02:54.864956098 +0000 UTC m=+2635.086991640" watchObservedRunningTime="2026-10-31 01:02:54.865854817 +0000 UTC m=+2635.087890360"
Oct 31 01:03:04.827316 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:04.827284    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl"
Oct 31 01:03:05.851062 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:05.849863    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr"
Oct 31 01:03:15.294766 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:03:15.294727    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:03:25.525186 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:25.523756    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": read tcp 10.133.0.2:34486->10.133.0.180:15021: read: connection reset by peer" start-of-body=
Oct 31 01:03:25.525186 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:25.523829    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz" podUID="13ff4194-5ef3-4f4d-b6e7-4cb5df757371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": read tcp 10.133.0.2:34486->10.133.0.180:15021: read: connection reset by peer"
Oct 31 01:03:26.114716 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:26.114671    4327 generic.go:356] "Generic (PLEG): container finished" podID="13ff4194-5ef3-4f4d-b6e7-4cb5df757371" containerID="fb69bc630de7bea5dd4698a8804940d4e0b4314d8bba5a78937eabfc882e2f05" exitCode=0
Oct 31 01:03:26.114935 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:26.114752    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz" event={"ID":"13ff4194-5ef3-4f4d-b6e7-4cb5df757371","Type":"ContainerDied","Data":"fb69bc630de7bea5dd4698a8804940d4e0b4314d8bba5a78937eabfc882e2f05"}
Oct 31 01:03:26.115248 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:26.115230    4327 scope.go:122] "RemoveContainer" containerID="fb69bc630de7bea5dd4698a8804940d4e0b4314d8bba5a78937eabfc882e2f05"
Oct 31 01:03:26.117118 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:26.117089    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:03:26.117200 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:26.117160    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:03:26.117200 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:26.117194    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:03:27.131348 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:27.131305    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz" event={"ID":"13ff4194-5ef3-4f4d-b6e7-4cb5df757371","Type":"ContainerStarted","Data":"1b262e5ace803963b4ac324e64dbd0383d1eed6fba2b7dfe66bce7c67d7eddd1"}
Oct 31 01:03:27.685363 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:27.685323    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:03:27.685650 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:27.685628    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 31 01:03:27.685737 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:27.685709    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz" podUID="13ff4194-5ef3-4f4d-b6e7-4cb5df757371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 31 01:03:28.686294 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:28.686259    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 31 01:03:28.686722 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:28.686313    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz" podUID="13ff4194-5ef3-4f4d-b6e7-4cb5df757371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 31 01:03:29.685800 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:29.685764    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 31 01:03:29.685985 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:29.685825    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz" podUID="13ff4194-5ef3-4f4d-b6e7-4cb5df757371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 31 01:03:30.686023 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:30.685984    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 31 01:03:30.686423 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:30.686043    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz" podUID="13ff4194-5ef3-4f4d-b6e7-4cb5df757371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 31 01:03:31.685597 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:31.685557    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 31 01:03:31.685777 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:31.685628    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz" podUID="13ff4194-5ef3-4f4d-b6e7-4cb5df757371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 31 01:03:32.689722 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:32.688528    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 31 01:03:32.689722 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:32.688602    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz" podUID="13ff4194-5ef3-4f4d-b6e7-4cb5df757371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 31 01:03:33.690322 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:33.690293    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:03:33.690844 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:33.690821    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:03:33.691485 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:33.691462    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:03:36.396335 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:36.396292    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t"]
Oct 31 01:03:36.402123 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:36.402100    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t"
Oct 31 01:03:36.434840 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:36.434802    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-b8fd5a\""
Oct 31 01:03:36.439450 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:36.439419    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-b8fd5a-dockercfg-sd488\""
Oct 31 01:03:36.565731 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:36.545761    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/852c332c-c02b-4ccc-b4ef-8733645922c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t\" (UID: \"852c332c-c02b-4ccc-b4ef-8733645922c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t"
Oct 31 01:03:36.565731 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:36.545929    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw4pw\" (UniqueName: \"kubernetes.io/projected/852c332c-c02b-4ccc-b4ef-8733645922c9-kube-api-access-cw4pw\") pod \"payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t\" (UID: \"852c332c-c02b-4ccc-b4ef-8733645922c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t"
Oct 31 01:03:36.610748 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:36.591106    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t"]
Oct 31 01:03:36.611806 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:36.611778    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97"]
Oct 31 01:03:36.619699 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:36.619659    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97"
Oct 31 01:03:36.628031 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:36.626060    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97"]
Oct 31 01:03:36.647906 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:36.647831    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw4pw\" (UniqueName: \"kubernetes.io/projected/852c332c-c02b-4ccc-b4ef-8733645922c9-kube-api-access-cw4pw\") pod \"payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t\" (UID: \"852c332c-c02b-4ccc-b4ef-8733645922c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t"
Oct 31 01:03:36.655510 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:36.655450    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/852c332c-c02b-4ccc-b4ef-8733645922c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t\" (UID: \"852c332c-c02b-4ccc-b4ef-8733645922c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t"
Oct 31 01:03:36.657635 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:36.657608    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/852c332c-c02b-4ccc-b4ef-8733645922c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t\" (UID: \"852c332c-c02b-4ccc-b4ef-8733645922c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t"
Oct 31 01:03:36.686752 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:36.686262    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw4pw\" (UniqueName: \"kubernetes.io/projected/852c332c-c02b-4ccc-b4ef-8733645922c9-kube-api-access-cw4pw\") pod \"payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t\" (UID: \"852c332c-c02b-4ccc-b4ef-8733645922c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t"
Oct 31 01:03:36.719337 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:36.719304    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t"
Oct 31 01:03:36.757899 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:36.757866    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96e41c65-75e1-4501-b965-14fa687c68c9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97\" (UID: \"96e41c65-75e1-4501-b965-14fa687c68c9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97"
Oct 31 01:03:36.758061 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:36.757910    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7gsf\" (UniqueName: \"kubernetes.io/projected/96e41c65-75e1-4501-b965-14fa687c68c9-kube-api-access-z7gsf\") pod \"payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97\" (UID: \"96e41c65-75e1-4501-b965-14fa687c68c9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97"
Oct 31 01:03:36.859849 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:36.859809    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96e41c65-75e1-4501-b965-14fa687c68c9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97\" (UID: \"96e41c65-75e1-4501-b965-14fa687c68c9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97"
Oct 31 01:03:36.860025 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:36.859861    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7gsf\" (UniqueName: \"kubernetes.io/projected/96e41c65-75e1-4501-b965-14fa687c68c9-kube-api-access-z7gsf\") pod \"payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97\" (UID: \"96e41c65-75e1-4501-b965-14fa687c68c9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97"
Oct 31 01:03:36.860589 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:36.860569    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96e41c65-75e1-4501-b965-14fa687c68c9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97\" (UID: \"96e41c65-75e1-4501-b965-14fa687c68c9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97"
Oct 31 01:03:36.894205 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:36.894169    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7gsf\" (UniqueName: \"kubernetes.io/projected/96e41c65-75e1-4501-b965-14fa687c68c9-kube-api-access-z7gsf\") pod \"payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97\" (UID: \"96e41c65-75e1-4501-b965-14fa687c68c9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97"
Oct 31 01:03:36.944814 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:36.944736    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97"
Oct 31 01:03:37.046537 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:37.046507    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t"]
Oct 31 01:03:37.237356 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:37.237276    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t" event={"ID":"852c332c-c02b-4ccc-b4ef-8733645922c9","Type":"ContainerStarted","Data":"ad07de1e5863ffa3fa213ca07b65e6868c2974aa5977ff4dab513c593e426427"}
Oct 31 01:03:37.237356 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:37.237332    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t" event={"ID":"852c332c-c02b-4ccc-b4ef-8733645922c9","Type":"ContainerStarted","Data":"bea9f43228facc142dea313d29660002cdcb0a2f8122da2c845531b91a9a2037"}
Oct 31 01:03:37.237552 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:37.237359    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t"
Oct 31 01:03:37.250364 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:03:37.250335    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e41c65_75e1_4501_b965_14fa687c68c9.slice/crio-8b2e26a103a7e253374f24df1d115dc0169882fb3270616a24c64343589d6a60 WatchSource:0}: Error finding container 8b2e26a103a7e253374f24df1d115dc0169882fb3270616a24c64343589d6a60: Status 404 returned error can't find the container with id 8b2e26a103a7e253374f24df1d115dc0169882fb3270616a24c64343589d6a60
Oct 31 01:03:37.251458 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:37.251436    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97"]
Oct 31 01:03:37.280870 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:37.280825    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t" podStartSLOduration=1.280812464 podStartE2EDuration="1.280812464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:37.278960804 +0000 UTC m=+2677.500996346" watchObservedRunningTime="2026-10-31 01:03:37.280812464 +0000 UTC m=+2677.502848005"
Oct 31 01:03:38.261702 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:38.254745    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97" event={"ID":"96e41c65-75e1-4501-b965-14fa687c68c9","Type":"ContainerStarted","Data":"123afc8e585a785406de80a2f2a1de1cbb1f8b79e8f3508795ee8e4a5a303904"}
Oct 31 01:03:38.261702 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:38.254813    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97" event={"ID":"96e41c65-75e1-4501-b965-14fa687c68c9","Type":"ContainerStarted","Data":"8b2e26a103a7e253374f24df1d115dc0169882fb3270616a24c64343589d6a60"}
Oct 31 01:03:38.261702 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:38.254857    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97"
Oct 31 01:03:38.302674 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:38.302619    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97" podStartSLOduration=2.302604486 podStartE2EDuration="2.302604486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:38.301160552 +0000 UTC m=+2678.523196095" watchObservedRunningTime="2026-10-31 01:03:38.302604486 +0000 UTC m=+2678.524640077"
Oct 31 01:03:45.293756 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:03:45.293719    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:03:48.256829 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:48.256799    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t"
Oct 31 01:03:50.271118 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:03:50.271089    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97"
Oct 31 01:04:15.294277 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:04:15.294231    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:04:18.064307 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:18.064271    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm"]
Oct 31 01:04:22.351642 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:22.351545    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"]
Oct 31 01:04:22.353815 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:22.353767    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd" podUID="7b9657eb-33b5-4847-9ce7-39085e57f7ae" containerName="istio-proxy" containerID="cri-o://dc3ade288691c2bfaa1403c7bfa95e3c3ac0ece34574b479476086504926aad5" gracePeriod=30
Oct 31 01:04:23.065475 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:23.065411    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm" podUID="67a2779b-8ad1-40f8-b30c-d55209a1421d" containerName="payload-pre-processing" containerID="cri-o://c8368b84b77bdaca669587725e9b5760dc3a07d834ff8fd5f100d203054e23fc" gracePeriod=25
Oct 31 01:04:25.660644 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:25.660608    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm" podUID="67a2779b-8ad1-40f8-b30c-d55209a1421d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:04:26.483428 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:26.483387    4327 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:04:26.483623 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:26.483464    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd" podUID="7b9657eb-33b5-4847-9ce7-39085e57f7ae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:04:27.643918 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.643844    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 01:04:27.766401 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.766363    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istiod-ca-cert\") pod \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") "
Oct 31 01:04:27.766611 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.766437    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-credential-socket\") pod \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") "
Oct 31 01:04:27.766611 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.766514    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-envoy\") pod \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") "
Oct 31 01:04:27.766611 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.766594    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-token\" (UniqueName: \"kubernetes.io/projected/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-token\") pod \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") "
Oct 31 01:04:27.766817 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.766646    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-podinfo\") pod \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") "
Oct 31 01:04:27.767424 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.766884    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-workload-certs\") pod \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") "
Oct 31 01:04:27.767424 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.766889    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istiod-ca-cert" pod "7b9657eb-33b5-4847-9ce7-39085e57f7ae" (UID: "7b9657eb-33b5-4847-9ce7-39085e57f7ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:04:27.767424 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.766974    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-data\") pod \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") "
Oct 31 01:04:27.767424 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.767015    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b9657eb-33b5-4847-9ce7-39085e57f7ae-kube-api-access-5k6rh\" (UniqueName: \"kubernetes.io/projected/7b9657eb-33b5-4847-9ce7-39085e57f7ae-kube-api-access-5k6rh\") pod \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") "
Oct 31 01:04:27.767424 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.767100    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-credential-socket" pod "7b9657eb-33b5-4847-9ce7-39085e57f7ae" (UID: "7b9657eb-33b5-4847-9ce7-39085e57f7ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:04:27.767424 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.767112    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-workload-socket\") pod \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\" (UID: \"7b9657eb-33b5-4847-9ce7-39085e57f7ae\") "
Oct 31 01:04:27.767424 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.767327    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-data" pod "7b9657eb-33b5-4847-9ce7-39085e57f7ae" (UID: "7b9657eb-33b5-4847-9ce7-39085e57f7ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:04:27.767884 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.767459    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-workload-socket" pod "7b9657eb-33b5-4847-9ce7-39085e57f7ae" (UID: "7b9657eb-33b5-4847-9ce7-39085e57f7ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:04:27.767884 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.767250    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-workload-certs" pod "7b9657eb-33b5-4847-9ce7-39085e57f7ae" (UID: "7b9657eb-33b5-4847-9ce7-39085e57f7ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:04:27.767991 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.767937    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:27.767991 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.767959    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:27.767991 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.767975    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:27.767991 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.767988    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:27.768205 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.768003    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:27.769615 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.769586    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-token" pod "7b9657eb-33b5-4847-9ce7-39085e57f7ae" (UID: "7b9657eb-33b5-4847-9ce7-39085e57f7ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:04:27.771122 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.771094    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-podinfo" pod "7b9657eb-33b5-4847-9ce7-39085e57f7ae" (UID: "7b9657eb-33b5-4847-9ce7-39085e57f7ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:04:27.771233 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.771134    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b9657eb-33b5-4847-9ce7-39085e57f7ae-kube-api-access-5k6rh" pod "7b9657eb-33b5-4847-9ce7-39085e57f7ae" (UID: "7b9657eb-33b5-4847-9ce7-39085e57f7ae"). InnerVolumeSpecName "kube-api-access-5k6rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:04:27.771432 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.771415    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-envoy" pod "7b9657eb-33b5-4847-9ce7-39085e57f7ae" (UID: "7b9657eb-33b5-4847-9ce7-39085e57f7ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:04:27.790975 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.790945    4327 generic.go:356] "Generic (PLEG): container finished" podID="7b9657eb-33b5-4847-9ce7-39085e57f7ae" containerID="dc3ade288691c2bfaa1403c7bfa95e3c3ac0ece34574b479476086504926aad5" exitCode=0
Oct 31 01:04:27.791115 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.790995    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"
Oct 31 01:04:27.791115 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.791023    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd" event={"ID":"7b9657eb-33b5-4847-9ce7-39085e57f7ae","Type":"ContainerDied","Data":"dc3ade288691c2bfaa1403c7bfa95e3c3ac0ece34574b479476086504926aad5"}
Oct 31 01:04:27.791115 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.791081    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd" event={"ID":"7b9657eb-33b5-4847-9ce7-39085e57f7ae","Type":"ContainerDied","Data":"a48244fd20ac16db7a97ba010c43dea3f2f6c5d3d7a743326d6a57c18dfd52bf"}
Oct 31 01:04:27.791115 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.791111    4327 scope.go:122] "RemoveContainer" containerID="dc3ade288691c2bfaa1403c7bfa95e3c3ac0ece34574b479476086504926aad5"
Oct 31 01:04:27.805035 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.805011    4327 scope.go:122] "RemoveContainer" containerID="dc3ade288691c2bfaa1403c7bfa95e3c3ac0ece34574b479476086504926aad5"
Oct 31 01:04:27.805368 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:04:27.805335    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc3ade288691c2bfaa1403c7bfa95e3c3ac0ece34574b479476086504926aad5\": container with ID starting with dc3ade288691c2bfaa1403c7bfa95e3c3ac0ece34574b479476086504926aad5 not found: ID does not exist" containerID="dc3ade288691c2bfaa1403c7bfa95e3c3ac0ece34574b479476086504926aad5"
Oct 31 01:04:27.805465 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.805386    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc3ade288691c2bfaa1403c7bfa95e3c3ac0ece34574b479476086504926aad5"} err="failed to get container status \"dc3ade288691c2bfaa1403c7bfa95e3c3ac0ece34574b479476086504926aad5\": rpc error: code = NotFound desc = could not find container \"dc3ade288691c2bfaa1403c7bfa95e3c3ac0ece34574b479476086504926aad5\": container with ID starting with dc3ade288691c2bfaa1403c7bfa95e3c3ac0ece34574b479476086504926aad5 not found: ID does not exist"
Oct 31 01:04:27.861798 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.861769    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"]
Oct 31 01:04:27.869384 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.869351    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:27.869384 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.869380    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:27.869603 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.869393    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b9657eb-33b5-4847-9ce7-39085e57f7ae-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:27.869603 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.869407    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k6rh\" (UniqueName: \"kubernetes.io/projected/7b9657eb-33b5-4847-9ce7-39085e57f7ae-kube-api-access-5k6rh\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:27.873863 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.873840    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-d7cbd"]
Oct 31 01:04:27.923747 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:27.923649    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b9657eb-33b5-4847-9ce7-39085e57f7ae" path="/var/lib/kubelet/pods/7b9657eb-33b5-4847-9ce7-39085e57f7ae/volumes"
Oct 31 01:04:30.566430 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:30.566393    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw"]
Oct 31 01:04:30.567228 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:30.567211    4327 cpu_manager.go:414] "containerMap: removing container" podUID="7b9657eb-33b5-4847-9ce7-39085e57f7ae" containerName="istio-proxy"
Oct 31 01:04:30.567277 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:30.567229    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b9657eb-33b5-4847-9ce7-39085e57f7ae" containerName="istio-proxy"
Oct 31 01:04:30.567410 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:30.567401    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b9657eb-33b5-4847-9ce7-39085e57f7ae" containerName="istio-proxy"
Oct 31 01:04:30.572020 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:30.571993    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw"
Oct 31 01:04:30.607746 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:30.605333    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-328d61\""
Oct 31 01:04:30.627766 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:30.627103    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-328d61-dockercfg-qtpvp\""
Oct 31 01:04:30.648829 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:30.648800    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw"]
Oct 31 01:04:30.712359 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:30.712015    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc"]
Oct 31 01:04:30.722558 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:30.722522    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc"]
Oct 31 01:04:30.722753 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:30.722639    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc"
Oct 31 01:04:30.729434 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:30.729127    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5a254c-034c-4179-9e52-f3db86dbfd7e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw\" (UID: \"cf5a254c-034c-4179-9e52-f3db86dbfd7e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw"
Oct 31 01:04:30.729434 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:30.729332    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv65n\" (UniqueName: \"kubernetes.io/projected/cf5a254c-034c-4179-9e52-f3db86dbfd7e-kube-api-access-pv65n\") pod \"payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw\" (UID: \"cf5a254c-034c-4179-9e52-f3db86dbfd7e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw"
Oct 31 01:04:30.879910 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:30.878664    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/495d81aa-af2c-4a7c-832d-61dfb5f7531c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc\" (UID: \"495d81aa-af2c-4a7c-832d-61dfb5f7531c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc"
Oct 31 01:04:30.879910 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:30.878823    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv65n\" (UniqueName: \"kubernetes.io/projected/cf5a254c-034c-4179-9e52-f3db86dbfd7e-kube-api-access-pv65n\") pod \"payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw\" (UID: \"cf5a254c-034c-4179-9e52-f3db86dbfd7e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw"
Oct 31 01:04:30.879910 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:30.878997    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj9vc\" (UniqueName: \"kubernetes.io/projected/495d81aa-af2c-4a7c-832d-61dfb5f7531c-kube-api-access-qj9vc\") pod \"payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc\" (UID: \"495d81aa-af2c-4a7c-832d-61dfb5f7531c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc"
Oct 31 01:04:30.879910 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:30.879111    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5a254c-034c-4179-9e52-f3db86dbfd7e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw\" (UID: \"cf5a254c-034c-4179-9e52-f3db86dbfd7e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw"
Oct 31 01:04:30.880414 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:30.880173    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5a254c-034c-4179-9e52-f3db86dbfd7e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw\" (UID: \"cf5a254c-034c-4179-9e52-f3db86dbfd7e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw"
Oct 31 01:04:30.903848 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:30.903807    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv65n\" (UniqueName: \"kubernetes.io/projected/cf5a254c-034c-4179-9e52-f3db86dbfd7e-kube-api-access-pv65n\") pod \"payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw\" (UID: \"cf5a254c-034c-4179-9e52-f3db86dbfd7e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw"
Oct 31 01:04:30.933093 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:30.933056    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw"
Oct 31 01:04:30.948439 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:30.948410    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244"]
Oct 31 01:04:30.980741 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:30.980667    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj9vc\" (UniqueName: \"kubernetes.io/projected/495d81aa-af2c-4a7c-832d-61dfb5f7531c-kube-api-access-qj9vc\") pod \"payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc\" (UID: \"495d81aa-af2c-4a7c-832d-61dfb5f7531c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc"
Oct 31 01:04:30.981002 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:30.980981    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/495d81aa-af2c-4a7c-832d-61dfb5f7531c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc\" (UID: \"495d81aa-af2c-4a7c-832d-61dfb5f7531c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc"
Oct 31 01:04:30.981915 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:30.981888    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/495d81aa-af2c-4a7c-832d-61dfb5f7531c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc\" (UID: \"495d81aa-af2c-4a7c-832d-61dfb5f7531c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc"
Oct 31 01:04:30.982997 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:30.982967    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b"]
Oct 31 01:04:31.004896 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:31.004862    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj9vc\" (UniqueName: \"kubernetes.io/projected/495d81aa-af2c-4a7c-832d-61dfb5f7531c-kube-api-access-qj9vc\") pod \"payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc\" (UID: \"495d81aa-af2c-4a7c-832d-61dfb5f7531c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc"
Oct 31 01:04:31.085232 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:31.084779    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc"
Oct 31 01:04:31.301509 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:31.301478    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw"]
Oct 31 01:04:31.301642 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:04:31.301612    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf5a254c_034c_4179_9e52_f3db86dbfd7e.slice/crio-b7eb315a24db6f0e81d50f468599551c329345a970d5fccff4980ade9ef848ae WatchSource:0}: Error finding container b7eb315a24db6f0e81d50f468599551c329345a970d5fccff4980ade9ef848ae: Status 404 returned error can't find the container with id b7eb315a24db6f0e81d50f468599551c329345a970d5fccff4980ade9ef848ae
Oct 31 01:04:31.412138 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:04:31.412102    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod495d81aa_af2c_4a7c_832d_61dfb5f7531c.slice/crio-6f06699ade3342c9643ff8c02d6e0717f893de0a8fa0e0845c90512816f43f8e WatchSource:0}: Error finding container 6f06699ade3342c9643ff8c02d6e0717f893de0a8fa0e0845c90512816f43f8e: Status 404 returned error can't find the container with id 6f06699ade3342c9643ff8c02d6e0717f893de0a8fa0e0845c90512816f43f8e
Oct 31 01:04:31.414550 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:31.414528    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc"]
Oct 31 01:04:31.892947 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:31.892907    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc" event={"ID":"495d81aa-af2c-4a7c-832d-61dfb5f7531c","Type":"ContainerStarted","Data":"48db15f68c8bd634f2543fa35c81b973abf9a34d6833f61bd16b47540e2131ee"}
Oct 31 01:04:31.893397 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:31.893013    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc" event={"ID":"495d81aa-af2c-4a7c-832d-61dfb5f7531c","Type":"ContainerStarted","Data":"6f06699ade3342c9643ff8c02d6e0717f893de0a8fa0e0845c90512816f43f8e"}
Oct 31 01:04:31.893397 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:31.893061    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc"
Oct 31 01:04:31.894750 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:31.894723    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw" event={"ID":"cf5a254c-034c-4179-9e52-f3db86dbfd7e","Type":"ContainerStarted","Data":"54d4778fba44fe6e38fe4b370e05221c76b88185d21b2e28e73a93c030c310d6"}
Oct 31 01:04:31.894864 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:31.894765    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw" event={"ID":"cf5a254c-034c-4179-9e52-f3db86dbfd7e","Type":"ContainerStarted","Data":"b7eb315a24db6f0e81d50f468599551c329345a970d5fccff4980ade9ef848ae"}
Oct 31 01:04:31.894939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:31.894877    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw"
Oct 31 01:04:31.939936 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:31.939879    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc" podStartSLOduration=1.939864056 podStartE2EDuration="1.939864056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:31.936035224 +0000 UTC m=+2732.158070769" watchObservedRunningTime="2026-10-31 01:04:31.939864056 +0000 UTC m=+2732.161899597"
Oct 31 01:04:31.990607 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:31.990544    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw" podStartSLOduration=1.990529593 podStartE2EDuration="1.990529593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:31.98761988 +0000 UTC m=+2732.209655423" watchObservedRunningTime="2026-10-31 01:04:31.990529593 +0000 UTC m=+2732.212565164"
Oct 31 01:04:35.660982 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:35.660936    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm" podUID="67a2779b-8ad1-40f8-b30c-d55209a1421d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:04:35.949617 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:35.948863    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244" podUID="08dd4688-b889-43ab-bba9-58afcf509595" containerName="payload-processing" containerID="cri-o://e38b5fee287278953f936f2ce53f4be4692b283798d218b66ec6de757611e4ab" gracePeriod=25
Oct 31 01:04:35.990864 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:35.990810    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b" podUID="b271c9fe-0a7d-4b81-b145-4db3f776e136" containerName="payload-pre-processing" containerID="cri-o://5ff213c3a14eb61a4d1d7c884b58a7d8d7c60467c6f07a653baaa3eb9dec0aca" gracePeriod=25
Oct 31 01:04:39.259951 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:39.259910    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b" podUID="b271c9fe-0a7d-4b81-b145-4db3f776e136" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:04:39.260371 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:39.260001    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244" podUID="08dd4688-b889-43ab-bba9-58afcf509595" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:04:42.906193 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:42.906120    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw"
Oct 31 01:04:42.906612 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:42.906563    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc"
Oct 31 01:04:43.446233 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:43.446206    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm"
Oct 31 01:04:43.538461 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:43.538138    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67a2779b-8ad1-40f8-b30c-d55209a1421d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a2779b-8ad1-40f8-b30c-d55209a1421d-plugins-config-volume\") pod \"67a2779b-8ad1-40f8-b30c-d55209a1421d\" (UID: \"67a2779b-8ad1-40f8-b30c-d55209a1421d\") "
Oct 31 01:04:43.538631 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:43.538550    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a2779b-8ad1-40f8-b30c-d55209a1421d-plugins-config-volume" pod "67a2779b-8ad1-40f8-b30c-d55209a1421d" (UID: "67a2779b-8ad1-40f8-b30c-d55209a1421d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:04:43.538750 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:43.538645    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67a2779b-8ad1-40f8-b30c-d55209a1421d-kube-api-access-gk7m5\" (UniqueName: \"kubernetes.io/projected/67a2779b-8ad1-40f8-b30c-d55209a1421d-kube-api-access-gk7m5\") pod \"67a2779b-8ad1-40f8-b30c-d55209a1421d\" (UID: \"67a2779b-8ad1-40f8-b30c-d55209a1421d\") "
Oct 31 01:04:43.539560 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:43.539535    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a2779b-8ad1-40f8-b30c-d55209a1421d-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:43.541059 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:43.541031    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a2779b-8ad1-40f8-b30c-d55209a1421d-kube-api-access-gk7m5" pod "67a2779b-8ad1-40f8-b30c-d55209a1421d" (UID: "67a2779b-8ad1-40f8-b30c-d55209a1421d"). InnerVolumeSpecName "kube-api-access-gk7m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:04:43.641005 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:43.640961    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk7m5\" (UniqueName: \"kubernetes.io/projected/67a2779b-8ad1-40f8-b30c-d55209a1421d-kube-api-access-gk7m5\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:43.993809 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:43.993765    4327 generic.go:356] "Generic (PLEG): container finished" podID="67a2779b-8ad1-40f8-b30c-d55209a1421d" containerID="c8368b84b77bdaca669587725e9b5760dc3a07d834ff8fd5f100d203054e23fc" exitCode=0
Oct 31 01:04:43.994283 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:43.993818    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm"
Oct 31 01:04:43.994283 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:43.993849    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm" event={"ID":"67a2779b-8ad1-40f8-b30c-d55209a1421d","Type":"ContainerDied","Data":"c8368b84b77bdaca669587725e9b5760dc3a07d834ff8fd5f100d203054e23fc"}
Oct 31 01:04:43.994283 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:43.993906    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm" event={"ID":"67a2779b-8ad1-40f8-b30c-d55209a1421d","Type":"ContainerDied","Data":"0b1fa31d4c739bfe685f8bb981adbe2cdd1ab0953eb77105030c83d427265fd4"}
Oct 31 01:04:43.994283 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:43.993928    4327 scope.go:122] "RemoveContainer" containerID="c8368b84b77bdaca669587725e9b5760dc3a07d834ff8fd5f100d203054e23fc"
Oct 31 01:04:44.005343 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:44.005320    4327 scope.go:122] "RemoveContainer" containerID="c8368b84b77bdaca669587725e9b5760dc3a07d834ff8fd5f100d203054e23fc"
Oct 31 01:04:44.005726 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:04:44.005660    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8368b84b77bdaca669587725e9b5760dc3a07d834ff8fd5f100d203054e23fc\": container with ID starting with c8368b84b77bdaca669587725e9b5760dc3a07d834ff8fd5f100d203054e23fc not found: ID does not exist" containerID="c8368b84b77bdaca669587725e9b5760dc3a07d834ff8fd5f100d203054e23fc"
Oct 31 01:04:44.005726 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:44.005707    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8368b84b77bdaca669587725e9b5760dc3a07d834ff8fd5f100d203054e23fc"} err="failed to get container status \"c8368b84b77bdaca669587725e9b5760dc3a07d834ff8fd5f100d203054e23fc\": rpc error: code = NotFound desc = could not find container \"c8368b84b77bdaca669587725e9b5760dc3a07d834ff8fd5f100d203054e23fc\": container with ID starting with c8368b84b77bdaca669587725e9b5760dc3a07d834ff8fd5f100d203054e23fc not found: ID does not exist"
Oct 31 01:04:44.054652 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:44.054616    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm"]
Oct 31 01:04:44.061169 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:44.061137    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9de7f5-85bdf664d7-kxrlm"]
Oct 31 01:04:45.294727 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:04:45.294667    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:04:45.923631 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:45.923592    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67a2779b-8ad1-40f8-b30c-d55209a1421d" path="/var/lib/kubelet/pods/67a2779b-8ad1-40f8-b30c-d55209a1421d/volumes"
Oct 31 01:04:49.259333 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:49.259290    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b" podUID="b271c9fe-0a7d-4b81-b145-4db3f776e136" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:04:49.259333 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:49.259308    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244" podUID="08dd4688-b889-43ab-bba9-58afcf509595" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:04:51.293928 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:51.293890    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx"]
Oct 31 01:04:51.294333 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:51.294115    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx" podUID="a3722996-be27-491c-9ee4-62733e3071a9" containerName="maas-api" containerID="cri-o://7fb36aff183db63cc71b805343608f54fbc3a015539c4f9646e22c555fabb8ad" gracePeriod=30
Oct 31 01:04:51.648610 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:51.648580    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx"
Oct 31 01:04:51.759701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:51.754197    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3722996-be27-491c-9ee4-62733e3071a9-kube-api-access-kpk7f\" (UniqueName: \"kubernetes.io/projected/a3722996-be27-491c-9ee4-62733e3071a9-kube-api-access-kpk7f\") pod \"a3722996-be27-491c-9ee4-62733e3071a9\" (UID: \"a3722996-be27-491c-9ee4-62733e3071a9\") "
Oct 31 01:04:51.759701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:51.754273    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3722996-be27-491c-9ee4-62733e3071a9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3722996-be27-491c-9ee4-62733e3071a9-metrics-certs\") pod \"a3722996-be27-491c-9ee4-62733e3071a9\" (UID: \"a3722996-be27-491c-9ee4-62733e3071a9\") "
Oct 31 01:04:51.759701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:51.754324    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3722996-be27-491c-9ee4-62733e3071a9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3722996-be27-491c-9ee4-62733e3071a9-maas-api-tls\") pod \"a3722996-be27-491c-9ee4-62733e3071a9\" (UID: \"a3722996-be27-491c-9ee4-62733e3071a9\") "
Oct 31 01:04:51.759701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:51.754401    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3722996-be27-491c-9ee4-62733e3071a9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3722996-be27-491c-9ee4-62733e3071a9-openshift-service-ca\") pod \"a3722996-be27-491c-9ee4-62733e3071a9\" (UID: \"a3722996-be27-491c-9ee4-62733e3071a9\") "
Oct 31 01:04:51.759701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:51.757927    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3722996-be27-491c-9ee4-62733e3071a9-openshift-service-ca" pod "a3722996-be27-491c-9ee4-62733e3071a9" (UID: "a3722996-be27-491c-9ee4-62733e3071a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:04:51.774855 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:51.768623    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3722996-be27-491c-9ee4-62733e3071a9-maas-api-tls" pod "a3722996-be27-491c-9ee4-62733e3071a9" (UID: "a3722996-be27-491c-9ee4-62733e3071a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:04:51.786900 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:51.783890    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3722996-be27-491c-9ee4-62733e3071a9-metrics-certs" pod "a3722996-be27-491c-9ee4-62733e3071a9" (UID: "a3722996-be27-491c-9ee4-62733e3071a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:04:51.794274 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:51.794220    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3722996-be27-491c-9ee4-62733e3071a9-kube-api-access-kpk7f" pod "a3722996-be27-491c-9ee4-62733e3071a9" (UID: "a3722996-be27-491c-9ee4-62733e3071a9"). InnerVolumeSpecName "kube-api-access-kpk7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:04:51.862399 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:51.862306    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3722996-be27-491c-9ee4-62733e3071a9-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:51.873751 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:51.863153    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpk7f\" (UniqueName: \"kubernetes.io/projected/a3722996-be27-491c-9ee4-62733e3071a9-kube-api-access-kpk7f\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:51.873751 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:51.863345    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3722996-be27-491c-9ee4-62733e3071a9-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:51.873751 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:51.863363    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3722996-be27-491c-9ee4-62733e3071a9-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:52.059014 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:52.058980    4327 generic.go:356] "Generic (PLEG): container finished" podID="a3722996-be27-491c-9ee4-62733e3071a9" containerID="7fb36aff183db63cc71b805343608f54fbc3a015539c4f9646e22c555fabb8ad" exitCode=0
Oct 31 01:04:52.059188 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:52.059024    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx"
Oct 31 01:04:52.059188 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:52.059060    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx" event={"ID":"a3722996-be27-491c-9ee4-62733e3071a9","Type":"ContainerDied","Data":"7fb36aff183db63cc71b805343608f54fbc3a015539c4f9646e22c555fabb8ad"}
Oct 31 01:04:52.059188 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:52.059117    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx" event={"ID":"a3722996-be27-491c-9ee4-62733e3071a9","Type":"ContainerDied","Data":"54dcc9e4a8b3f931b7a8f103a61d87bb5a29ffecd1f5c523ac7c7a6fa03fe7b6"}
Oct 31 01:04:52.059188 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:52.059140    4327 scope.go:122] "RemoveContainer" containerID="7fb36aff183db63cc71b805343608f54fbc3a015539c4f9646e22c555fabb8ad"
Oct 31 01:04:52.069870 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:52.069845    4327 scope.go:122] "RemoveContainer" containerID="7fb36aff183db63cc71b805343608f54fbc3a015539c4f9646e22c555fabb8ad"
Oct 31 01:04:52.070176 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:04:52.070157    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb36aff183db63cc71b805343608f54fbc3a015539c4f9646e22c555fabb8ad\": container with ID starting with 7fb36aff183db63cc71b805343608f54fbc3a015539c4f9646e22c555fabb8ad not found: ID does not exist" containerID="7fb36aff183db63cc71b805343608f54fbc3a015539c4f9646e22c555fabb8ad"
Oct 31 01:04:52.070261 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:52.070186    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb36aff183db63cc71b805343608f54fbc3a015539c4f9646e22c555fabb8ad"} err="failed to get container status \"7fb36aff183db63cc71b805343608f54fbc3a015539c4f9646e22c555fabb8ad\": rpc error: code = NotFound desc = could not find container \"7fb36aff183db63cc71b805343608f54fbc3a015539c4f9646e22c555fabb8ad\": container with ID starting with 7fb36aff183db63cc71b805343608f54fbc3a015539c4f9646e22c555fabb8ad not found: ID does not exist"
Oct 31 01:04:52.118037 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:52.117945    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx"]
Oct 31 01:04:52.121725 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:52.121701    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9de7f5-689b79567-wfpvx"]
Oct 31 01:04:52.208360 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:52.208322    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"]
Oct 31 01:04:52.208551 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:52.208529    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk" podUID="242be892-3b92-44cb-a8ef-17f8ac581b4b" containerName="istio-proxy" containerID="cri-o://fec6de9c040ea38c58718a3e1ed0cd8ae305e29eca555895eeb1d7db0df3c80e" gracePeriod=30
Oct 31 01:04:52.858617 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:52.858581    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"]
Oct 31 01:04:53.922921 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:53.922888    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3722996-be27-491c-9ee4-62733e3071a9" path="/var/lib/kubelet/pods/a3722996-be27-491c-9ee4-62733e3071a9/volumes"
Oct 31 01:04:56.099008 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:56.098948    4327 generic.go:356] "Generic (PLEG): container finished" podID="08dd4688-b889-43ab-bba9-58afcf509595" containerID="e38b5fee287278953f936f2ce53f4be4692b283798d218b66ec6de757611e4ab" exitCode=0
Oct 31 01:04:56.099362 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:56.099026    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244" event={"ID":"08dd4688-b889-43ab-bba9-58afcf509595","Type":"ContainerDied","Data":"e38b5fee287278953f936f2ce53f4be4692b283798d218b66ec6de757611e4ab"}
Oct 31 01:04:56.287378 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:56.287355    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244"
Oct 31 01:04:56.305475 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:56.305449    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b"
Oct 31 01:04:56.391094 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:56.391061    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08dd4688-b889-43ab-bba9-58afcf509595-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08dd4688-b889-43ab-bba9-58afcf509595-plugins-config-volume\") pod \"08dd4688-b889-43ab-bba9-58afcf509595\" (UID: \"08dd4688-b889-43ab-bba9-58afcf509595\") "
Oct 31 01:04:56.391285 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:56.391108    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b271c9fe-0a7d-4b81-b145-4db3f776e136-kube-api-access-dmnhr\" (UniqueName: \"kubernetes.io/projected/b271c9fe-0a7d-4b81-b145-4db3f776e136-kube-api-access-dmnhr\") pod \"b271c9fe-0a7d-4b81-b145-4db3f776e136\" (UID: \"b271c9fe-0a7d-4b81-b145-4db3f776e136\") "
Oct 31 01:04:56.391285 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:56.391195    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b271c9fe-0a7d-4b81-b145-4db3f776e136-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b271c9fe-0a7d-4b81-b145-4db3f776e136-plugins-config-volume\") pod \"b271c9fe-0a7d-4b81-b145-4db3f776e136\" (UID: \"b271c9fe-0a7d-4b81-b145-4db3f776e136\") "
Oct 31 01:04:56.391432 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:56.391419    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08dd4688-b889-43ab-bba9-58afcf509595-kube-api-access-gqqxk\" (UniqueName: \"kubernetes.io/projected/08dd4688-b889-43ab-bba9-58afcf509595-kube-api-access-gqqxk\") pod \"08dd4688-b889-43ab-bba9-58afcf509595\" (UID: \"08dd4688-b889-43ab-bba9-58afcf509595\") "
Oct 31 01:04:56.391543 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:56.391517    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08dd4688-b889-43ab-bba9-58afcf509595-plugins-config-volume" pod "08dd4688-b889-43ab-bba9-58afcf509595" (UID: "08dd4688-b889-43ab-bba9-58afcf509595"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:04:56.391618 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:56.391594    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b271c9fe-0a7d-4b81-b145-4db3f776e136-plugins-config-volume" pod "b271c9fe-0a7d-4b81-b145-4db3f776e136" (UID: "b271c9fe-0a7d-4b81-b145-4db3f776e136"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:04:56.392019 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:56.391992    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08dd4688-b889-43ab-bba9-58afcf509595-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:56.392019 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:56.392018    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b271c9fe-0a7d-4b81-b145-4db3f776e136-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:56.393412 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:56.393386    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b271c9fe-0a7d-4b81-b145-4db3f776e136-kube-api-access-dmnhr" pod "b271c9fe-0a7d-4b81-b145-4db3f776e136" (UID: "b271c9fe-0a7d-4b81-b145-4db3f776e136"). InnerVolumeSpecName "kube-api-access-dmnhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:04:56.393579 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:56.393556    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08dd4688-b889-43ab-bba9-58afcf509595-kube-api-access-gqqxk" pod "08dd4688-b889-43ab-bba9-58afcf509595" (UID: "08dd4688-b889-43ab-bba9-58afcf509595"). InnerVolumeSpecName "kube-api-access-gqqxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:04:56.493966 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:56.493926    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqqxk\" (UniqueName: \"kubernetes.io/projected/08dd4688-b889-43ab-bba9-58afcf509595-kube-api-access-gqqxk\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:56.493966 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:56.493955    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmnhr\" (UniqueName: \"kubernetes.io/projected/b271c9fe-0a7d-4b81-b145-4db3f776e136-kube-api-access-dmnhr\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:57.107455 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.107418    4327 generic.go:356] "Generic (PLEG): container finished" podID="b271c9fe-0a7d-4b81-b145-4db3f776e136" containerID="5ff213c3a14eb61a4d1d7c884b58a7d8d7c60467c6f07a653baaa3eb9dec0aca" exitCode=0
Oct 31 01:04:57.107946 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.107465    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b" event={"ID":"b271c9fe-0a7d-4b81-b145-4db3f776e136","Type":"ContainerDied","Data":"5ff213c3a14eb61a4d1d7c884b58a7d8d7c60467c6f07a653baaa3eb9dec0aca"}
Oct 31 01:04:57.107946 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.107525    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b" event={"ID":"b271c9fe-0a7d-4b81-b145-4db3f776e136","Type":"ContainerDied","Data":"fae356fb466cbfbdc79212eace2cbbce0abb5a24687a103fc20d9ec1a0ee4677"}
Oct 31 01:04:57.107946 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.107547    4327 scope.go:122] "RemoveContainer" containerID="5ff213c3a14eb61a4d1d7c884b58a7d8d7c60467c6f07a653baaa3eb9dec0aca"
Oct 31 01:04:57.107946 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.107474    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b"
Oct 31 01:04:57.109439 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.109412    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244" event={"ID":"08dd4688-b889-43ab-bba9-58afcf509595","Type":"ContainerDied","Data":"b5fb7927f9dda5acc4143c188e928014d7af2437e177ce0e67b4870225e4833e"}
Oct 31 01:04:57.109439 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.109430    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244"
Oct 31 01:04:57.120234 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.120216    4327 scope.go:122] "RemoveContainer" containerID="5ff213c3a14eb61a4d1d7c884b58a7d8d7c60467c6f07a653baaa3eb9dec0aca"
Oct 31 01:04:57.120505 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:04:57.120486    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ff213c3a14eb61a4d1d7c884b58a7d8d7c60467c6f07a653baaa3eb9dec0aca\": container with ID starting with 5ff213c3a14eb61a4d1d7c884b58a7d8d7c60467c6f07a653baaa3eb9dec0aca not found: ID does not exist" containerID="5ff213c3a14eb61a4d1d7c884b58a7d8d7c60467c6f07a653baaa3eb9dec0aca"
Oct 31 01:04:57.120561 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.120511    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff213c3a14eb61a4d1d7c884b58a7d8d7c60467c6f07a653baaa3eb9dec0aca"} err="failed to get container status \"5ff213c3a14eb61a4d1d7c884b58a7d8d7c60467c6f07a653baaa3eb9dec0aca\": rpc error: code = NotFound desc = could not find container \"5ff213c3a14eb61a4d1d7c884b58a7d8d7c60467c6f07a653baaa3eb9dec0aca\": container with ID starting with 5ff213c3a14eb61a4d1d7c884b58a7d8d7c60467c6f07a653baaa3eb9dec0aca not found: ID does not exist"
Oct 31 01:04:57.120561 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.120530    4327 scope.go:122] "RemoveContainer" containerID="e38b5fee287278953f936f2ce53f4be4692b283798d218b66ec6de757611e4ab"
Oct 31 01:04:57.173585 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.173553    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244"]
Oct 31 01:04:57.186890 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.186854    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b87c62-6b8656d99c-ph244"]
Oct 31 01:04:57.223861 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.223827    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b"]
Oct 31 01:04:57.227566 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.227513    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b87c62-7745bdc775-cd27b"]
Oct 31 01:04:57.528462 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.528436    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 01:04:57.611160 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.611128    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/242be892-3b92-44cb-a8ef-17f8ac581b4b-kube-api-access-6k2c9\" (UniqueName: \"kubernetes.io/projected/242be892-3b92-44cb-a8ef-17f8ac581b4b-kube-api-access-6k2c9\") pod \"242be892-3b92-44cb-a8ef-17f8ac581b4b\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") "
Oct 31 01:04:57.611363 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.611182    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-token\" (UniqueName: \"kubernetes.io/projected/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-token\") pod \"242be892-3b92-44cb-a8ef-17f8ac581b4b\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") "
Oct 31 01:04:57.611363 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.611286    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-podinfo\") pod \"242be892-3b92-44cb-a8ef-17f8ac581b4b\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") "
Oct 31 01:04:57.611363 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.611334    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-credential-socket\") pod \"242be892-3b92-44cb-a8ef-17f8ac581b4b\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") "
Oct 31 01:04:57.611530 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.611455    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-workload-certs\") pod \"242be892-3b92-44cb-a8ef-17f8ac581b4b\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") "
Oct 31 01:04:57.611530 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.611511    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-envoy\") pod \"242be892-3b92-44cb-a8ef-17f8ac581b4b\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") "
Oct 31 01:04:57.611642 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.611550    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-workload-socket\") pod \"242be892-3b92-44cb-a8ef-17f8ac581b4b\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") "
Oct 31 01:04:57.611642 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.611581    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/242be892-3b92-44cb-a8ef-17f8ac581b4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242be892-3b92-44cb-a8ef-17f8ac581b4b-istiod-ca-cert\") pod \"242be892-3b92-44cb-a8ef-17f8ac581b4b\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") "
Oct 31 01:04:57.611766 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.611637    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-credential-socket" pod "242be892-3b92-44cb-a8ef-17f8ac581b4b" (UID: "242be892-3b92-44cb-a8ef-17f8ac581b4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:04:57.611766 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.611661    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-data\") pod \"242be892-3b92-44cb-a8ef-17f8ac581b4b\" (UID: \"242be892-3b92-44cb-a8ef-17f8ac581b4b\") "
Oct 31 01:04:57.611937 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.611918    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-data" pod "242be892-3b92-44cb-a8ef-17f8ac581b4b" (UID: "242be892-3b92-44cb-a8ef-17f8ac581b4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:04:57.612106 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.612089    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-workload-socket" pod "242be892-3b92-44cb-a8ef-17f8ac581b4b" (UID: "242be892-3b92-44cb-a8ef-17f8ac581b4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:04:57.612335 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.612318    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242be892-3b92-44cb-a8ef-17f8ac581b4b-istiod-ca-cert" pod "242be892-3b92-44cb-a8ef-17f8ac581b4b" (UID: "242be892-3b92-44cb-a8ef-17f8ac581b4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:04:57.612564 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.612540    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-workload-certs" pod "242be892-3b92-44cb-a8ef-17f8ac581b4b" (UID: "242be892-3b92-44cb-a8ef-17f8ac581b4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:04:57.612714 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.612670    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:57.612714 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.612709    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242be892-3b92-44cb-a8ef-17f8ac581b4b-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:57.612856 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.612723    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:57.612856 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.612735    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:57.612856 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.612750    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:57.613770 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.613744    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-token" pod "242be892-3b92-44cb-a8ef-17f8ac581b4b" (UID: "242be892-3b92-44cb-a8ef-17f8ac581b4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:04:57.614017 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.613998    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-podinfo" pod "242be892-3b92-44cb-a8ef-17f8ac581b4b" (UID: "242be892-3b92-44cb-a8ef-17f8ac581b4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:04:57.614017 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.614003    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242be892-3b92-44cb-a8ef-17f8ac581b4b-kube-api-access-6k2c9" pod "242be892-3b92-44cb-a8ef-17f8ac581b4b" (UID: "242be892-3b92-44cb-a8ef-17f8ac581b4b"). InnerVolumeSpecName "kube-api-access-6k2c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:04:57.614922 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.614898    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-envoy" pod "242be892-3b92-44cb-a8ef-17f8ac581b4b" (UID: "242be892-3b92-44cb-a8ef-17f8ac581b4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:04:57.754510 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.753042    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:57.754510 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.753088    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:57.754510 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.753105    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242be892-3b92-44cb-a8ef-17f8ac581b4b-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:57.754510 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.753122    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k2c9\" (UniqueName: \"kubernetes.io/projected/242be892-3b92-44cb-a8ef-17f8ac581b4b-kube-api-access-6k2c9\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:04:57.924476 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.924443    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08dd4688-b889-43ab-bba9-58afcf509595" path="/var/lib/kubelet/pods/08dd4688-b889-43ab-bba9-58afcf509595/volumes"
Oct 31 01:04:57.924952 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:57.924938    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b271c9fe-0a7d-4b81-b145-4db3f776e136" path="/var/lib/kubelet/pods/b271c9fe-0a7d-4b81-b145-4db3f776e136/volumes"
Oct 31 01:04:58.120142 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:58.120107    4327 generic.go:356] "Generic (PLEG): container finished" podID="242be892-3b92-44cb-a8ef-17f8ac581b4b" containerID="fec6de9c040ea38c58718a3e1ed0cd8ae305e29eca555895eeb1d7db0df3c80e" exitCode=0
Oct 31 01:04:58.120626 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:58.120159    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"
Oct 31 01:04:58.120626 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:58.120194    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk" event={"ID":"242be892-3b92-44cb-a8ef-17f8ac581b4b","Type":"ContainerDied","Data":"fec6de9c040ea38c58718a3e1ed0cd8ae305e29eca555895eeb1d7db0df3c80e"}
Oct 31 01:04:58.120626 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:58.120249    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk" event={"ID":"242be892-3b92-44cb-a8ef-17f8ac581b4b","Type":"ContainerDied","Data":"cc89b27ca47b1bd5370be0d499bee2069e29f7dfe452e6bf5241973d3bec8782"}
Oct 31 01:04:58.120626 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:58.120278    4327 scope.go:122] "RemoveContainer" containerID="fec6de9c040ea38c58718a3e1ed0cd8ae305e29eca555895eeb1d7db0df3c80e"
Oct 31 01:04:58.131105 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:58.131082    4327 scope.go:122] "RemoveContainer" containerID="352a138edc15db7342152a37d38e4c6ff57b41b971fe361b2276fa3ee8960229"
Oct 31 01:04:58.141490 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:58.141466    4327 scope.go:122] "RemoveContainer" containerID="fec6de9c040ea38c58718a3e1ed0cd8ae305e29eca555895eeb1d7db0df3c80e"
Oct 31 01:04:58.141903 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:04:58.141877    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fec6de9c040ea38c58718a3e1ed0cd8ae305e29eca555895eeb1d7db0df3c80e\": container with ID starting with fec6de9c040ea38c58718a3e1ed0cd8ae305e29eca555895eeb1d7db0df3c80e not found: ID does not exist" containerID="fec6de9c040ea38c58718a3e1ed0cd8ae305e29eca555895eeb1d7db0df3c80e"
Oct 31 01:04:58.142000 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:58.141912    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fec6de9c040ea38c58718a3e1ed0cd8ae305e29eca555895eeb1d7db0df3c80e"} err="failed to get container status \"fec6de9c040ea38c58718a3e1ed0cd8ae305e29eca555895eeb1d7db0df3c80e\": rpc error: code = NotFound desc = could not find container \"fec6de9c040ea38c58718a3e1ed0cd8ae305e29eca555895eeb1d7db0df3c80e\": container with ID starting with fec6de9c040ea38c58718a3e1ed0cd8ae305e29eca555895eeb1d7db0df3c80e not found: ID does not exist"
Oct 31 01:04:58.142000 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:58.141929    4327 scope.go:122] "RemoveContainer" containerID="352a138edc15db7342152a37d38e4c6ff57b41b971fe361b2276fa3ee8960229"
Oct 31 01:04:58.142235 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:04:58.142218    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"352a138edc15db7342152a37d38e4c6ff57b41b971fe361b2276fa3ee8960229\": container with ID starting with 352a138edc15db7342152a37d38e4c6ff57b41b971fe361b2276fa3ee8960229 not found: ID does not exist" containerID="352a138edc15db7342152a37d38e4c6ff57b41b971fe361b2276fa3ee8960229"
Oct 31 01:04:58.142288 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:58.142239    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"352a138edc15db7342152a37d38e4c6ff57b41b971fe361b2276fa3ee8960229"} err="failed to get container status \"352a138edc15db7342152a37d38e4c6ff57b41b971fe361b2276fa3ee8960229\": rpc error: code = NotFound desc = could not find container \"352a138edc15db7342152a37d38e4c6ff57b41b971fe361b2276fa3ee8960229\": container with ID starting with 352a138edc15db7342152a37d38e4c6ff57b41b971fe361b2276fa3ee8960229 not found: ID does not exist"
Oct 31 01:04:58.188273 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:58.188238    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"]
Oct 31 01:04:58.200425 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:58.200395    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b87c62-openshift-default-79f485ccf5-qnlfk"]
Oct 31 01:04:59.923304 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:04:59.923275    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="242be892-3b92-44cb-a8ef-17f8ac581b4b" path="/var/lib/kubelet/pods/242be892-3b92-44cb-a8ef-17f8ac581b4b/volumes"
Oct 31 01:05:00.253212 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:00.253132    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"]
Oct 31 01:05:00.253383 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:00.253358    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt" podUID="b046f39f-ed7f-414e-aedf-be830e648883" containerName="istio-proxy" containerID="cri-o://c59b84d2b1f421651193a5d0b1166f9361ebb7403a662a1cb5311ab22d982e8e" gracePeriod=30
Oct 31 01:05:03.623780 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:03.623745    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf"]
Oct 31 01:05:03.659972 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:03.659940    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8"]
Oct 31 01:05:05.533545 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.533480    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 01:05:05.596725 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.596099    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-workload-socket\") pod \"b046f39f-ed7f-414e-aedf-be830e648883\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") "
Oct 31 01:05:05.596725 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.596165    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-credential-socket\") pod \"b046f39f-ed7f-414e-aedf-be830e648883\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") "
Oct 31 01:05:05.596725 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.596198    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b046f39f-ed7f-414e-aedf-be830e648883-istio-token\" (UniqueName: \"kubernetes.io/projected/b046f39f-ed7f-414e-aedf-be830e648883-istio-token\") pod \"b046f39f-ed7f-414e-aedf-be830e648883\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") "
Oct 31 01:05:05.596725 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.596236    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-istio-data\") pod \"b046f39f-ed7f-414e-aedf-be830e648883\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") "
Oct 31 01:05:05.596725 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.596323    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b046f39f-ed7f-414e-aedf-be830e648883-kube-api-access-5zxxs\" (UniqueName: \"kubernetes.io/projected/b046f39f-ed7f-414e-aedf-be830e648883-kube-api-access-5zxxs\") pod \"b046f39f-ed7f-414e-aedf-be830e648883\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") "
Oct 31 01:05:05.596725 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.596392    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-workload-socket" pod "b046f39f-ed7f-414e-aedf-be830e648883" (UID: "b046f39f-ed7f-414e-aedf-be830e648883"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:05:05.596725 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.596595    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b046f39f-ed7f-414e-aedf-be830e648883-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b046f39f-ed7f-414e-aedf-be830e648883-istiod-ca-cert\") pod \"b046f39f-ed7f-414e-aedf-be830e648883\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") "
Oct 31 01:05:05.597396 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.596669    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-istio-envoy\") pod \"b046f39f-ed7f-414e-aedf-be830e648883\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") "
Oct 31 01:05:05.597396 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.596878    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-workload-certs\") pod \"b046f39f-ed7f-414e-aedf-be830e648883\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") "
Oct 31 01:05:05.597396 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.596933    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b046f39f-ed7f-414e-aedf-be830e648883-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b046f39f-ed7f-414e-aedf-be830e648883-istio-podinfo\") pod \"b046f39f-ed7f-414e-aedf-be830e648883\" (UID: \"b046f39f-ed7f-414e-aedf-be830e648883\") "
Oct 31 01:05:05.597396 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.597371    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b046f39f-ed7f-414e-aedf-be830e648883-istiod-ca-cert" pod "b046f39f-ed7f-414e-aedf-be830e648883" (UID: "b046f39f-ed7f-414e-aedf-be830e648883"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:05.597917 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.597706    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b046f39f-ed7f-414e-aedf-be830e648883-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:05.597917 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.597732    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:05.598108 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.598030    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-istio-data" pod "b046f39f-ed7f-414e-aedf-be830e648883" (UID: "b046f39f-ed7f-414e-aedf-be830e648883"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:05:05.598377 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.598338    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-credential-socket" pod "b046f39f-ed7f-414e-aedf-be830e648883" (UID: "b046f39f-ed7f-414e-aedf-be830e648883"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:05:05.598499 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.598391    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-workload-certs" pod "b046f39f-ed7f-414e-aedf-be830e648883" (UID: "b046f39f-ed7f-414e-aedf-be830e648883"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:05:05.599471 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.599446    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b046f39f-ed7f-414e-aedf-be830e648883-istio-token" pod "b046f39f-ed7f-414e-aedf-be830e648883" (UID: "b046f39f-ed7f-414e-aedf-be830e648883"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:05.599792 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.599766    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b046f39f-ed7f-414e-aedf-be830e648883-istio-podinfo" pod "b046f39f-ed7f-414e-aedf-be830e648883" (UID: "b046f39f-ed7f-414e-aedf-be830e648883"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:05:05.601057 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.601029    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-istio-envoy" pod "b046f39f-ed7f-414e-aedf-be830e648883" (UID: "b046f39f-ed7f-414e-aedf-be830e648883"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:05:05.601148 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.601095    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b046f39f-ed7f-414e-aedf-be830e648883-kube-api-access-5zxxs" pod "b046f39f-ed7f-414e-aedf-be830e648883" (UID: "b046f39f-ed7f-414e-aedf-be830e648883"). InnerVolumeSpecName "kube-api-access-5zxxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:05.699582 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.699550    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:05.699582 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.699579    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b046f39f-ed7f-414e-aedf-be830e648883-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:05.699837 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.699591    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:05.699837 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.699602    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zxxs\" (UniqueName: \"kubernetes.io/projected/b046f39f-ed7f-414e-aedf-be830e648883-kube-api-access-5zxxs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:05.699837 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.699610    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:05.699837 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.699620    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b046f39f-ed7f-414e-aedf-be830e648883-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:05.699837 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:05.699628    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b046f39f-ed7f-414e-aedf-be830e648883-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:06.187510 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:06.187472    4327 generic.go:356] "Generic (PLEG): container finished" podID="b046f39f-ed7f-414e-aedf-be830e648883" containerID="c59b84d2b1f421651193a5d0b1166f9361ebb7403a662a1cb5311ab22d982e8e" exitCode=0
Oct 31 01:05:06.187510 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:06.187515    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt" event={"ID":"b046f39f-ed7f-414e-aedf-be830e648883","Type":"ContainerDied","Data":"c59b84d2b1f421651193a5d0b1166f9361ebb7403a662a1cb5311ab22d982e8e"}
Oct 31 01:05:06.187791 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:06.187538    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"
Oct 31 01:05:06.187791 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:06.187556    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt" event={"ID":"b046f39f-ed7f-414e-aedf-be830e648883","Type":"ContainerDied","Data":"cc70f6dbd31a41ba36c92477b6f13c857f5a3bcebcc3fcb8dc87d31ba9bc53bc"}
Oct 31 01:05:06.187791 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:06.187579    4327 scope.go:122] "RemoveContainer" containerID="c59b84d2b1f421651193a5d0b1166f9361ebb7403a662a1cb5311ab22d982e8e"
Oct 31 01:05:06.198792 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:06.198768    4327 scope.go:122] "RemoveContainer" containerID="5984429b31752f91e808ec4c27cdac411d637af501d998cac60b16cb03a39e36"
Oct 31 01:05:06.208999 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:06.208973    4327 scope.go:122] "RemoveContainer" containerID="c59b84d2b1f421651193a5d0b1166f9361ebb7403a662a1cb5311ab22d982e8e"
Oct 31 01:05:06.209286 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:05:06.209262    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c59b84d2b1f421651193a5d0b1166f9361ebb7403a662a1cb5311ab22d982e8e\": container with ID starting with c59b84d2b1f421651193a5d0b1166f9361ebb7403a662a1cb5311ab22d982e8e not found: ID does not exist" containerID="c59b84d2b1f421651193a5d0b1166f9361ebb7403a662a1cb5311ab22d982e8e"
Oct 31 01:05:06.209362 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:06.209300    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c59b84d2b1f421651193a5d0b1166f9361ebb7403a662a1cb5311ab22d982e8e"} err="failed to get container status \"c59b84d2b1f421651193a5d0b1166f9361ebb7403a662a1cb5311ab22d982e8e\": rpc error: code = NotFound desc = could not find container \"c59b84d2b1f421651193a5d0b1166f9361ebb7403a662a1cb5311ab22d982e8e\": container with ID starting with c59b84d2b1f421651193a5d0b1166f9361ebb7403a662a1cb5311ab22d982e8e not found: ID does not exist"
Oct 31 01:05:06.209362 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:06.209318    4327 scope.go:122] "RemoveContainer" containerID="5984429b31752f91e808ec4c27cdac411d637af501d998cac60b16cb03a39e36"
Oct 31 01:05:06.209593 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:05:06.209573    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5984429b31752f91e808ec4c27cdac411d637af501d998cac60b16cb03a39e36\": container with ID starting with 5984429b31752f91e808ec4c27cdac411d637af501d998cac60b16cb03a39e36 not found: ID does not exist" containerID="5984429b31752f91e808ec4c27cdac411d637af501d998cac60b16cb03a39e36"
Oct 31 01:05:06.209637 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:06.209601    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5984429b31752f91e808ec4c27cdac411d637af501d998cac60b16cb03a39e36"} err="failed to get container status \"5984429b31752f91e808ec4c27cdac411d637af501d998cac60b16cb03a39e36\": rpc error: code = NotFound desc = could not find container \"5984429b31752f91e808ec4c27cdac411d637af501d998cac60b16cb03a39e36\": container with ID starting with 5984429b31752f91e808ec4c27cdac411d637af501d998cac60b16cb03a39e36 not found: ID does not exist"
Oct 31 01:05:06.267034 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:06.267003    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"]
Oct 31 01:05:06.279201 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:06.279163    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-mwqkt"]
Oct 31 01:05:07.887126 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:07.887062    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5" podUID="c3b9b681-9675-4c15-81d4-0fed4b4b2d63" containerName="main" containerID="cri-o://ae457bec7ea24817b44c755840ac69747682150e980b41e62f8d124783257a01" gracePeriod=45
Oct 31 01:05:07.891967 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:07.891924    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5" podUID="c3b9b681-9675-4c15-81d4-0fed4b4b2d63" containerName="main" probeResult="failure" output="Get \"https://10.133.0.165:8000/ready\": EOF"
Oct 31 01:05:07.923660 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:07.923626    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b046f39f-ed7f-414e-aedf-be830e648883" path="/var/lib/kubelet/pods/b046f39f-ed7f-414e-aedf-be830e648883/volumes"
Oct 31 01:05:08.177933 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.177902    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"
Oct 31 01:05:08.208897 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.208860    4327 generic.go:356] "Generic (PLEG): container finished" podID="c3b9b681-9675-4c15-81d4-0fed4b4b2d63" containerID="ae457bec7ea24817b44c755840ac69747682150e980b41e62f8d124783257a01" exitCode=0
Oct 31 01:05:08.209083 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.208921    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"
Oct 31 01:05:08.209083 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.208948    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5" event={"ID":"c3b9b681-9675-4c15-81d4-0fed4b4b2d63","Type":"ContainerDied","Data":"ae457bec7ea24817b44c755840ac69747682150e980b41e62f8d124783257a01"}
Oct 31 01:05:08.209083 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.209001    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5" event={"ID":"c3b9b681-9675-4c15-81d4-0fed4b4b2d63","Type":"ContainerDied","Data":"f95c26e1006b74ff88e46089b8dedec009323effc050a6ce87680b6ecfb32af2"}
Oct 31 01:05:08.209083 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.209025    4327 scope.go:122] "RemoveContainer" containerID="ae457bec7ea24817b44c755840ac69747682150e980b41e62f8d124783257a01"
Oct 31 01:05:08.219413 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.219390    4327 scope.go:122] "RemoveContainer" containerID="ae457bec7ea24817b44c755840ac69747682150e980b41e62f8d124783257a01"
Oct 31 01:05:08.219755 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:05:08.219734    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae457bec7ea24817b44c755840ac69747682150e980b41e62f8d124783257a01\": container with ID starting with ae457bec7ea24817b44c755840ac69747682150e980b41e62f8d124783257a01 not found: ID does not exist" containerID="ae457bec7ea24817b44c755840ac69747682150e980b41e62f8d124783257a01"
Oct 31 01:05:08.219826 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.219766    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae457bec7ea24817b44c755840ac69747682150e980b41e62f8d124783257a01"} err="failed to get container status \"ae457bec7ea24817b44c755840ac69747682150e980b41e62f8d124783257a01\": rpc error: code = NotFound desc = could not find container \"ae457bec7ea24817b44c755840ac69747682150e980b41e62f8d124783257a01\": container with ID starting with ae457bec7ea24817b44c755840ac69747682150e980b41e62f8d124783257a01 not found: ID does not exist"
Oct 31 01:05:08.236398 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.236361    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-home\" (UniqueName: \"kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-home\") pod \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\" (UID: \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\") "
Oct 31 01:05:08.236569 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.236471    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-tmp-dir\") pod \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\" (UID: \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\") "
Oct 31 01:05:08.236569 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.236511    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-model-cache\") pod \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\" (UID: \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\") "
Oct 31 01:05:08.236724 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.236604    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-tls-certs\" (UniqueName: \"kubernetes.io/secret/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-tls-certs\") pod \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\" (UID: \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\") "
Oct 31 01:05:08.236724 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.236634    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-kube-api-access-zv8wd\" (UniqueName: \"kubernetes.io/projected/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-kube-api-access-zv8wd\") pod \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\" (UID: \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\") "
Oct 31 01:05:08.236724 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.236659    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-home" pod "c3b9b681-9675-4c15-81d4-0fed4b4b2d63" (UID: "c3b9b681-9675-4c15-81d4-0fed4b4b2d63"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:05:08.236724 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.236694    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-dshm\") pod \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\" (UID: \"c3b9b681-9675-4c15-81d4-0fed4b4b2d63\") "
Oct 31 01:05:08.236955 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.236778    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-tmp-dir" pod "c3b9b681-9675-4c15-81d4-0fed4b4b2d63" (UID: "c3b9b681-9675-4c15-81d4-0fed4b4b2d63"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:05:08.237151 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.237124    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-model-cache" pod "c3b9b681-9675-4c15-81d4-0fed4b4b2d63" (UID: "c3b9b681-9675-4c15-81d4-0fed4b4b2d63"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:05:08.237516 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.237487    4327 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-home\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:08.237623 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.237521    4327 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-tmp-dir\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:08.237623 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.237536    4327 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-model-cache\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:08.239594 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.239565    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-dshm" pod "c3b9b681-9675-4c15-81d4-0fed4b4b2d63" (UID: "c3b9b681-9675-4c15-81d4-0fed4b4b2d63"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:05:08.239720 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.239593    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-tls-certs" pod "c3b9b681-9675-4c15-81d4-0fed4b4b2d63" (UID: "c3b9b681-9675-4c15-81d4-0fed4b4b2d63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:08.239720 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.239619    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-kube-api-access-zv8wd" pod "c3b9b681-9675-4c15-81d4-0fed4b4b2d63" (UID: "c3b9b681-9675-4c15-81d4-0fed4b4b2d63"). InnerVolumeSpecName "kube-api-access-zv8wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:08.338571 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.338526    4327 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-tls-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:08.338571 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.338559    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv8wd\" (UniqueName: \"kubernetes.io/projected/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-kube-api-access-zv8wd\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:08.338571 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.338569    4327 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3b9b681-9675-4c15-81d4-0fed4b4b2d63-dshm\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:08.559968 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.559928    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"]
Oct 31 01:05:08.571752 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.571718    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-72120c/test-model-72120c-kserve-7696d9f9d5-d2sh5"]
Oct 31 01:05:08.630404 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.626829    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf" podUID="cf355564-5ea9-4149-9540-d01964674903" containerName="payload-processing" containerID="cri-o://f7710cac8afed708d11351d308206a1467d68be7db6fb7734881fc91e94629f6" gracePeriod=25
Oct 31 01:05:08.661137 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:08.661091    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8" podUID="57e3b4e5-4f46-4c99-a260-3beb556431c7" containerName="payload-pre-processing" containerID="cri-o://40a46409a803826b1ff0f52d090826c434c2c15eeb9b89e6ac58a7c17dd8712c" gracePeriod=25
Oct 31 01:05:09.922994 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:09.922963    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3b9b681-9675-4c15-81d4-0fed4b4b2d63" path="/var/lib/kubelet/pods/c3b9b681-9675-4c15-81d4-0fed4b4b2d63/volumes"
Oct 31 01:05:10.406426 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:10.406385    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8" podUID="57e3b4e5-4f46-4c99-a260-3beb556431c7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:05:11.483338 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.483297    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf" podUID="cf355564-5ea9-4149-9540-d01964674903" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:05:11.632545 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.632507    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm"]
Oct 31 01:05:11.633530 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633499    4327 cpu_manager.go:414] "containerMap: removing container" podUID="08dd4688-b889-43ab-bba9-58afcf509595" containerName="payload-processing"
Oct 31 01:05:11.633530 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633526    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08dd4688-b889-43ab-bba9-58afcf509595" containerName="payload-processing"
Oct 31 01:05:11.633713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633550    4327 cpu_manager.go:414] "containerMap: removing container" podUID="b046f39f-ed7f-414e-aedf-be830e648883" containerName="istio-proxy"
Oct 31 01:05:11.633713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633555    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b046f39f-ed7f-414e-aedf-be830e648883" containerName="istio-proxy"
Oct 31 01:05:11.633713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633568    4327 cpu_manager.go:414] "containerMap: removing container" podUID="b046f39f-ed7f-414e-aedf-be830e648883" containerName="istio-proxy"
Oct 31 01:05:11.633713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633574    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b046f39f-ed7f-414e-aedf-be830e648883" containerName="istio-proxy"
Oct 31 01:05:11.633713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633601    4327 cpu_manager.go:414] "containerMap: removing container" podUID="242be892-3b92-44cb-a8ef-17f8ac581b4b" containerName="istio-proxy"
Oct 31 01:05:11.633713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633606    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="242be892-3b92-44cb-a8ef-17f8ac581b4b" containerName="istio-proxy"
Oct 31 01:05:11.633713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633612    4327 cpu_manager.go:414] "containerMap: removing container" podUID="b271c9fe-0a7d-4b81-b145-4db3f776e136" containerName="payload-pre-processing"
Oct 31 01:05:11.633713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633616    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b271c9fe-0a7d-4b81-b145-4db3f776e136" containerName="payload-pre-processing"
Oct 31 01:05:11.633713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633628    4327 cpu_manager.go:414] "containerMap: removing container" podUID="a3722996-be27-491c-9ee4-62733e3071a9" containerName="maas-api"
Oct 31 01:05:11.633713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633633    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3722996-be27-491c-9ee4-62733e3071a9" containerName="maas-api"
Oct 31 01:05:11.633713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633640    4327 cpu_manager.go:414] "containerMap: removing container" podUID="242be892-3b92-44cb-a8ef-17f8ac581b4b" containerName="istio-proxy"
Oct 31 01:05:11.633713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633644    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="242be892-3b92-44cb-a8ef-17f8ac581b4b" containerName="istio-proxy"
Oct 31 01:05:11.633713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633667    4327 cpu_manager.go:414] "containerMap: removing container" podUID="c3b9b681-9675-4c15-81d4-0fed4b4b2d63" containerName="main"
Oct 31 01:05:11.633713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633690    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3b9b681-9675-4c15-81d4-0fed4b4b2d63" containerName="main"
Oct 31 01:05:11.633713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633699    4327 cpu_manager.go:414] "containerMap: removing container" podUID="67a2779b-8ad1-40f8-b30c-d55209a1421d" containerName="payload-pre-processing"
Oct 31 01:05:11.633713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633705    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67a2779b-8ad1-40f8-b30c-d55209a1421d" containerName="payload-pre-processing"
Oct 31 01:05:11.634210 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633834    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3b9b681-9675-4c15-81d4-0fed4b4b2d63" containerName="main"
Oct 31 01:05:11.634210 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633850    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="67a2779b-8ad1-40f8-b30c-d55209a1421d" containerName="payload-pre-processing"
Oct 31 01:05:11.634210 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633859    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="08dd4688-b889-43ab-bba9-58afcf509595" containerName="payload-processing"
Oct 31 01:05:11.634210 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633868    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="b046f39f-ed7f-414e-aedf-be830e648883" containerName="istio-proxy"
Oct 31 01:05:11.634210 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633876    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="b046f39f-ed7f-414e-aedf-be830e648883" containerName="istio-proxy"
Oct 31 01:05:11.634210 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633885    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="242be892-3b92-44cb-a8ef-17f8ac581b4b" containerName="istio-proxy"
Oct 31 01:05:11.634210 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633892    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="b271c9fe-0a7d-4b81-b145-4db3f776e136" containerName="payload-pre-processing"
Oct 31 01:05:11.634210 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633897    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3722996-be27-491c-9ee4-62733e3071a9" containerName="maas-api"
Oct 31 01:05:11.634210 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.633903    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="242be892-3b92-44cb-a8ef-17f8ac581b4b" containerName="istio-proxy"
Oct 31 01:05:11.639011 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.638991    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm"
Oct 31 01:05:11.665955 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.665913    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-328d61-serving-cert\""
Oct 31 01:05:11.680752 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.680716    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm"]
Oct 31 01:05:11.683567 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.683525    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld2km\" (UniqueName: \"kubernetes.io/projected/39aa5706-7923-41cb-a27e-e2034f07e687-kube-api-access-ld2km\") pod \"maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm\" (UID: \"39aa5706-7923-41cb-a27e-e2034f07e687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm"
Oct 31 01:05:11.683779 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.683596    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39aa5706-7923-41cb-a27e-e2034f07e687-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm\" (UID: \"39aa5706-7923-41cb-a27e-e2034f07e687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm"
Oct 31 01:05:11.683779 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.683740    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39aa5706-7923-41cb-a27e-e2034f07e687-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm\" (UID: \"39aa5706-7923-41cb-a27e-e2034f07e687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm"
Oct 31 01:05:11.683931 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.683876    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39aa5706-7923-41cb-a27e-e2034f07e687-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm\" (UID: \"39aa5706-7923-41cb-a27e-e2034f07e687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm"
Oct 31 01:05:11.785929 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.785846    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39aa5706-7923-41cb-a27e-e2034f07e687-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm\" (UID: \"39aa5706-7923-41cb-a27e-e2034f07e687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm"
Oct 31 01:05:11.786085 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.785969    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39aa5706-7923-41cb-a27e-e2034f07e687-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm\" (UID: \"39aa5706-7923-41cb-a27e-e2034f07e687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm"
Oct 31 01:05:11.786128 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:05:11.786113    4327 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-serving-cert: secret "maas-api-e2e-shared-b-w5-328d61-serving-cert" not found
Oct 31 01:05:11.786183 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:05:11.786173    4327 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39aa5706-7923-41cb-a27e-e2034f07e687-maas-api-tls podName:39aa5706-7923-41cb-a27e-e2034f07e687 nodeName:}" failed. No retries permitted until 2026-10-31 01:05:12.286157475 +0000 UTC m=+2772.508192999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/39aa5706-7923-41cb-a27e-e2034f07e687-maas-api-tls") pod "maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm" (UID: "39aa5706-7923-41cb-a27e-e2034f07e687") : secret "maas-api-e2e-shared-b-w5-328d61-serving-cert" not found
Oct 31 01:05:11.786244 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.786120    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld2km\" (UniqueName: \"kubernetes.io/projected/39aa5706-7923-41cb-a27e-e2034f07e687-kube-api-access-ld2km\") pod \"maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm\" (UID: \"39aa5706-7923-41cb-a27e-e2034f07e687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm"
Oct 31 01:05:11.786304 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.786271    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39aa5706-7923-41cb-a27e-e2034f07e687-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm\" (UID: \"39aa5706-7923-41cb-a27e-e2034f07e687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm"
Oct 31 01:05:11.786902 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.786878    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39aa5706-7923-41cb-a27e-e2034f07e687-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm\" (UID: \"39aa5706-7923-41cb-a27e-e2034f07e687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm"
Oct 31 01:05:11.788514 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.788488    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39aa5706-7923-41cb-a27e-e2034f07e687-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm\" (UID: \"39aa5706-7923-41cb-a27e-e2034f07e687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm"
Oct 31 01:05:11.810266 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:11.810238    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld2km\" (UniqueName: \"kubernetes.io/projected/39aa5706-7923-41cb-a27e-e2034f07e687-kube-api-access-ld2km\") pod \"maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm\" (UID: \"39aa5706-7923-41cb-a27e-e2034f07e687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm"
Oct 31 01:05:12.293478 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:12.293441    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39aa5706-7923-41cb-a27e-e2034f07e687-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm\" (UID: \"39aa5706-7923-41cb-a27e-e2034f07e687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm"
Oct 31 01:05:12.296084 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:12.296063    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39aa5706-7923-41cb-a27e-e2034f07e687-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm\" (UID: \"39aa5706-7923-41cb-a27e-e2034f07e687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm"
Oct 31 01:05:12.554094 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:12.553994    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm"
Oct 31 01:05:12.912263 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:05:12.912232    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39aa5706_7923_41cb_a27e_e2034f07e687.slice/crio-8fa793bd1f459703de945dcb43a06e02153b09fa44916f658c34eea4435a67d7 WatchSource:0}: Error finding container 8fa793bd1f459703de945dcb43a06e02153b09fa44916f658c34eea4435a67d7: Status 404 returned error can't find the container with id 8fa793bd1f459703de945dcb43a06e02153b09fa44916f658c34eea4435a67d7
Oct 31 01:05:12.914836 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:12.914814    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm"]
Oct 31 01:05:13.271518 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:13.271435    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm" event={"ID":"39aa5706-7923-41cb-a27e-e2034f07e687","Type":"ContainerStarted","Data":"8fa793bd1f459703de945dcb43a06e02153b09fa44916f658c34eea4435a67d7"}
Oct 31 01:05:14.283754 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:14.283711    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm" event={"ID":"39aa5706-7923-41cb-a27e-e2034f07e687","Type":"ContainerStarted","Data":"0667c6a7ead02aa8c4a323adcffa340dbeef39917631fbd5533f858f20ea9661"}
Oct 31 01:05:14.284233 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:14.283845    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm"
Oct 31 01:05:14.329707 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:14.329640    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm" podStartSLOduration=2.894017963 podStartE2EDuration="3.329623797s" totalImagesPullingTime="435.605834ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:05:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:05:14.327449446 +0000 UTC m=+2774.549485002" watchObservedRunningTime="2026-10-31 01:05:14.329623797 +0000 UTC m=+2774.551659338"
Oct 31 01:05:14.408940 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:14.408877    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"]
Oct 31 01:05:15.295120 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:05:15.295080    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:05:20.301169 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:20.301136    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm"
Oct 31 01:05:20.409391 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:20.409343    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8" podUID="57e3b4e5-4f46-4c99-a260-3beb556431c7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:05:21.483271 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:21.483229    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf" podUID="cf355564-5ea9-4149-9540-d01964674903" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:05:25.470793 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:25.470758    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd"]
Oct 31 01:05:25.502371 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:25.502328    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79"]
Oct 31 01:05:28.953601 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:28.953572    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf"
Oct 31 01:05:28.995515 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:28.995473    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8"
Oct 31 01:05:29.011932 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.011888    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf355564-5ea9-4149-9540-d01964674903-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf355564-5ea9-4149-9540-d01964674903-plugins-config-volume\") pod \"cf355564-5ea9-4149-9540-d01964674903\" (UID: \"cf355564-5ea9-4149-9540-d01964674903\") "
Oct 31 01:05:29.012163 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.012086    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf355564-5ea9-4149-9540-d01964674903-kube-api-access-xhn8l\" (UniqueName: \"kubernetes.io/projected/cf355564-5ea9-4149-9540-d01964674903-kube-api-access-xhn8l\") pod \"cf355564-5ea9-4149-9540-d01964674903\" (UID: \"cf355564-5ea9-4149-9540-d01964674903\") "
Oct 31 01:05:29.012336 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.012309    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf355564-5ea9-4149-9540-d01964674903-plugins-config-volume" pod "cf355564-5ea9-4149-9540-d01964674903" (UID: "cf355564-5ea9-4149-9540-d01964674903"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:29.013004 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.012975    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf355564-5ea9-4149-9540-d01964674903-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:29.015073 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.015041    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf355564-5ea9-4149-9540-d01964674903-kube-api-access-xhn8l" pod "cf355564-5ea9-4149-9540-d01964674903" (UID: "cf355564-5ea9-4149-9540-d01964674903"). InnerVolumeSpecName "kube-api-access-xhn8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:29.114094 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.114057    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57e3b4e5-4f46-4c99-a260-3beb556431c7-kube-api-access-xtjq8\" (UniqueName: \"kubernetes.io/projected/57e3b4e5-4f46-4c99-a260-3beb556431c7-kube-api-access-xtjq8\") pod \"57e3b4e5-4f46-4c99-a260-3beb556431c7\" (UID: \"57e3b4e5-4f46-4c99-a260-3beb556431c7\") "
Oct 31 01:05:29.114267 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.114136    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57e3b4e5-4f46-4c99-a260-3beb556431c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57e3b4e5-4f46-4c99-a260-3beb556431c7-plugins-config-volume\") pod \"57e3b4e5-4f46-4c99-a260-3beb556431c7\" (UID: \"57e3b4e5-4f46-4c99-a260-3beb556431c7\") "
Oct 31 01:05:29.114549 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.114527    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57e3b4e5-4f46-4c99-a260-3beb556431c7-plugins-config-volume" pod "57e3b4e5-4f46-4c99-a260-3beb556431c7" (UID: "57e3b4e5-4f46-4c99-a260-3beb556431c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:29.114595 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.114584    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhn8l\" (UniqueName: \"kubernetes.io/projected/cf355564-5ea9-4149-9540-d01964674903-kube-api-access-xhn8l\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:29.116237 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.116220    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e3b4e5-4f46-4c99-a260-3beb556431c7-kube-api-access-xtjq8" pod "57e3b4e5-4f46-4c99-a260-3beb556431c7" (UID: "57e3b4e5-4f46-4c99-a260-3beb556431c7"). InnerVolumeSpecName "kube-api-access-xtjq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:29.215733 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.215628    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57e3b4e5-4f46-4c99-a260-3beb556431c7-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:29.215733 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.215660    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtjq8\" (UniqueName: \"kubernetes.io/projected/57e3b4e5-4f46-4c99-a260-3beb556431c7-kube-api-access-xtjq8\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:29.435823 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.435756    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz" podUID="e0a4bc42-fcaa-47bc-ad35-a654443342c1" containerName="main" containerID="cri-o://8577448987a5d663f5c93b2cc8bce055bba2a79f29a9cc91d8da476225bb6012" gracePeriod=45
Oct 31 01:05:29.447908 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.447874    4327 generic.go:356] "Generic (PLEG): container finished" podID="57e3b4e5-4f46-4c99-a260-3beb556431c7" containerID="40a46409a803826b1ff0f52d090826c434c2c15eeb9b89e6ac58a7c17dd8712c" exitCode=0
Oct 31 01:05:29.448074 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.447942    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8"
Oct 31 01:05:29.448074 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.447963    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8" event={"ID":"57e3b4e5-4f46-4c99-a260-3beb556431c7","Type":"ContainerDied","Data":"40a46409a803826b1ff0f52d090826c434c2c15eeb9b89e6ac58a7c17dd8712c"}
Oct 31 01:05:29.448214 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.448091    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8" event={"ID":"57e3b4e5-4f46-4c99-a260-3beb556431c7","Type":"ContainerDied","Data":"edd1d6492be78468f91a946c9bd9168633fe54906f66b4ef2146efcadf7358bc"}
Oct 31 01:05:29.448214 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.448130    4327 scope.go:122] "RemoveContainer" containerID="40a46409a803826b1ff0f52d090826c434c2c15eeb9b89e6ac58a7c17dd8712c"
Oct 31 01:05:29.449495 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.449477    4327 generic.go:356] "Generic (PLEG): container finished" podID="cf355564-5ea9-4149-9540-d01964674903" containerID="f7710cac8afed708d11351d308206a1467d68be7db6fb7734881fc91e94629f6" exitCode=0
Oct 31 01:05:29.449581 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.449545    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf"
Oct 31 01:05:29.449581 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.449560    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf" event={"ID":"cf355564-5ea9-4149-9540-d01964674903","Type":"ContainerDied","Data":"f7710cac8afed708d11351d308206a1467d68be7db6fb7734881fc91e94629f6"}
Oct 31 01:05:29.449669 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.449615    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf" event={"ID":"cf355564-5ea9-4149-9540-d01964674903","Type":"ContainerDied","Data":"00e41492e2ebad59685cb18e313a0d92ff805a708ead6bf3b00f727d981368ff"}
Oct 31 01:05:29.531731 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.531696    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf"]
Oct 31 01:05:29.544571 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.544539    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-01a117-849b4fc645-28vbf"]
Oct 31 01:05:29.549229 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.549197    4327 scope.go:122] "RemoveContainer" containerID="40a46409a803826b1ff0f52d090826c434c2c15eeb9b89e6ac58a7c17dd8712c"
Oct 31 01:05:29.549607 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:05:29.549578    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40a46409a803826b1ff0f52d090826c434c2c15eeb9b89e6ac58a7c17dd8712c\": container with ID starting with 40a46409a803826b1ff0f52d090826c434c2c15eeb9b89e6ac58a7c17dd8712c not found: ID does not exist" containerID="40a46409a803826b1ff0f52d090826c434c2c15eeb9b89e6ac58a7c17dd8712c"
Oct 31 01:05:29.549667 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.549622    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40a46409a803826b1ff0f52d090826c434c2c15eeb9b89e6ac58a7c17dd8712c"} err="failed to get container status \"40a46409a803826b1ff0f52d090826c434c2c15eeb9b89e6ac58a7c17dd8712c\": rpc error: code = NotFound desc = could not find container \"40a46409a803826b1ff0f52d090826c434c2c15eeb9b89e6ac58a7c17dd8712c\": container with ID starting with 40a46409a803826b1ff0f52d090826c434c2c15eeb9b89e6ac58a7c17dd8712c not found: ID does not exist"
Oct 31 01:05:29.549743 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.549666    4327 scope.go:122] "RemoveContainer" containerID="f7710cac8afed708d11351d308206a1467d68be7db6fb7734881fc91e94629f6"
Oct 31 01:05:29.559787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.559760    4327 scope.go:122] "RemoveContainer" containerID="f7710cac8afed708d11351d308206a1467d68be7db6fb7734881fc91e94629f6"
Oct 31 01:05:29.560062 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:05:29.560039    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7710cac8afed708d11351d308206a1467d68be7db6fb7734881fc91e94629f6\": container with ID starting with f7710cac8afed708d11351d308206a1467d68be7db6fb7734881fc91e94629f6 not found: ID does not exist" containerID="f7710cac8afed708d11351d308206a1467d68be7db6fb7734881fc91e94629f6"
Oct 31 01:05:29.560134 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.560071    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7710cac8afed708d11351d308206a1467d68be7db6fb7734881fc91e94629f6"} err="failed to get container status \"f7710cac8afed708d11351d308206a1467d68be7db6fb7734881fc91e94629f6\": rpc error: code = NotFound desc = could not find container \"f7710cac8afed708d11351d308206a1467d68be7db6fb7734881fc91e94629f6\": container with ID starting with f7710cac8afed708d11351d308206a1467d68be7db6fb7734881fc91e94629f6 not found: ID does not exist"
Oct 31 01:05:29.578853 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.578813    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8"]
Oct 31 01:05:29.586077 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.584673    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-01a117-96d7f8b99-4c9m8"]
Oct 31 01:05:29.735271 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.735245    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"
Oct 31 01:05:29.825037 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.824947    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-dshm\") pod \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\" (UID: \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\") "
Oct 31 01:05:29.825037 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.824999    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0a4bc42-fcaa-47bc-ad35-a654443342c1-tls-certs\" (UniqueName: \"kubernetes.io/secret/e0a4bc42-fcaa-47bc-ad35-a654443342c1-tls-certs\") pod \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\" (UID: \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\") "
Oct 31 01:05:29.825273 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.825117    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0a4bc42-fcaa-47bc-ad35-a654443342c1-kube-api-access-6987v\" (UniqueName: \"kubernetes.io/projected/e0a4bc42-fcaa-47bc-ad35-a654443342c1-kube-api-access-6987v\") pod \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\" (UID: \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\") "
Oct 31 01:05:29.825273 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.825182    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-tmp-dir\") pod \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\" (UID: \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\") "
Oct 31 01:05:29.825273 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.825264    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-model-cache\") pod \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\" (UID: \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\") "
Oct 31 01:05:29.825454 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.825301    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-home\" (UniqueName: \"kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-home\") pod \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\" (UID: \"e0a4bc42-fcaa-47bc-ad35-a654443342c1\") "
Oct 31 01:05:29.825708 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.825648    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-tmp-dir" pod "e0a4bc42-fcaa-47bc-ad35-a654443342c1" (UID: "e0a4bc42-fcaa-47bc-ad35-a654443342c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:05:29.825919 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.825780    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-model-cache" pod "e0a4bc42-fcaa-47bc-ad35-a654443342c1" (UID: "e0a4bc42-fcaa-47bc-ad35-a654443342c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:05:29.825919 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.825841    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-home" pod "e0a4bc42-fcaa-47bc-ad35-a654443342c1" (UID: "e0a4bc42-fcaa-47bc-ad35-a654443342c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:05:29.826275 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.826237    4327 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-tmp-dir\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:29.826275 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.826275    4327 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-model-cache\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:29.826456 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.826289    4327 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-home\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:29.827525 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.827492    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0a4bc42-fcaa-47bc-ad35-a654443342c1-tls-certs" pod "e0a4bc42-fcaa-47bc-ad35-a654443342c1" (UID: "e0a4bc42-fcaa-47bc-ad35-a654443342c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:29.827723 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.827669    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-dshm" pod "e0a4bc42-fcaa-47bc-ad35-a654443342c1" (UID: "e0a4bc42-fcaa-47bc-ad35-a654443342c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:05:29.827959 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.827936    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a4bc42-fcaa-47bc-ad35-a654443342c1-kube-api-access-6987v" pod "e0a4bc42-fcaa-47bc-ad35-a654443342c1" (UID: "e0a4bc42-fcaa-47bc-ad35-a654443342c1"). InnerVolumeSpecName "kube-api-access-6987v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:29.923287 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.923256    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57e3b4e5-4f46-4c99-a260-3beb556431c7" path="/var/lib/kubelet/pods/57e3b4e5-4f46-4c99-a260-3beb556431c7/volumes"
Oct 31 01:05:29.923752 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.923738    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf355564-5ea9-4149-9540-d01964674903" path="/var/lib/kubelet/pods/cf355564-5ea9-4149-9540-d01964674903/volumes"
Oct 31 01:05:29.927800 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.927771    4327 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0a4bc42-fcaa-47bc-ad35-a654443342c1-dshm\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:29.927800 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.927797    4327 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0a4bc42-fcaa-47bc-ad35-a654443342c1-tls-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:29.928011 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:29.927810    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6987v\" (UniqueName: \"kubernetes.io/projected/e0a4bc42-fcaa-47bc-ad35-a654443342c1-kube-api-access-6987v\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:30.484701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:30.481768    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd" podUID="9b0715eb-599c-4663-874c-b350f7a2910d" containerName="payload-processing" containerID="cri-o://add5fa3e5de12e3c452c9c4ed39c39ea47f54b10873a2db002995a724c2428c8" gracePeriod=25
Oct 31 01:05:30.511716 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:30.505734    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79" podUID="0f28a16d-689f-4baf-96ac-c2c9e7908888" containerName="payload-pre-processing" containerID="cri-o://5594c9583f5073f1d392ece714a8d3fe3ee0cafac91459c9942fc30e3d35d4ba" gracePeriod=25
Oct 31 01:05:30.527717 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:30.525652    4327 generic.go:356] "Generic (PLEG): container finished" podID="e0a4bc42-fcaa-47bc-ad35-a654443342c1" containerID="8577448987a5d663f5c93b2cc8bce055bba2a79f29a9cc91d8da476225bb6012" exitCode=0
Oct 31 01:05:30.527717 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:30.525825    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz" event={"ID":"e0a4bc42-fcaa-47bc-ad35-a654443342c1","Type":"ContainerDied","Data":"8577448987a5d663f5c93b2cc8bce055bba2a79f29a9cc91d8da476225bb6012"}
Oct 31 01:05:30.527717 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:30.525866    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz" event={"ID":"e0a4bc42-fcaa-47bc-ad35-a654443342c1","Type":"ContainerDied","Data":"523adf7b00bb2d2a98c637ff5e2f885c69c852181019849db9b9576cc4a4f449"}
Oct 31 01:05:30.527717 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:30.525889    4327 scope.go:122] "RemoveContainer" containerID="8577448987a5d663f5c93b2cc8bce055bba2a79f29a9cc91d8da476225bb6012"
Oct 31 01:05:30.527717 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:30.526012    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"
Oct 31 01:05:30.564361 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:30.564272    4327 scope.go:122] "RemoveContainer" containerID="8577448987a5d663f5c93b2cc8bce055bba2a79f29a9cc91d8da476225bb6012"
Oct 31 01:05:30.566082 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:05:30.566054    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8577448987a5d663f5c93b2cc8bce055bba2a79f29a9cc91d8da476225bb6012\": container with ID starting with 8577448987a5d663f5c93b2cc8bce055bba2a79f29a9cc91d8da476225bb6012 not found: ID does not exist" containerID="8577448987a5d663f5c93b2cc8bce055bba2a79f29a9cc91d8da476225bb6012"
Oct 31 01:05:30.566224 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:30.566090    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8577448987a5d663f5c93b2cc8bce055bba2a79f29a9cc91d8da476225bb6012"} err="failed to get container status \"8577448987a5d663f5c93b2cc8bce055bba2a79f29a9cc91d8da476225bb6012\": rpc error: code = NotFound desc = could not find container \"8577448987a5d663f5c93b2cc8bce055bba2a79f29a9cc91d8da476225bb6012\": container with ID starting with 8577448987a5d663f5c93b2cc8bce055bba2a79f29a9cc91d8da476225bb6012 not found: ID does not exist"
Oct 31 01:05:30.598420 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:30.598387    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"]
Oct 31 01:05:30.610880 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:30.610852    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-82b5e8/test-model-82b5e8-kserve-5cb569446c-7xngz"]
Oct 31 01:05:30.705534 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:30.705497    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd" podUID="9b0715eb-599c-4663-874c-b350f7a2910d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:05:30.705753 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:30.705512    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79" podUID="0f28a16d-689f-4baf-96ac-c2c9e7908888" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:05:31.922814 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:31.922782    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0a4bc42-fcaa-47bc-ad35-a654443342c1" path="/var/lib/kubelet/pods/e0a4bc42-fcaa-47bc-ad35-a654443342c1/volumes"
Oct 31 01:05:39.437451 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:39.437407    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"]
Oct 31 01:05:39.437894 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:39.437691    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz" podUID="9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3" containerName="istio-proxy" containerID="cri-o://643e90a2b6044dcd604705694292c07f068e0758c6f11dc689b1e4ef6d9a8604" gracePeriod=30
Oct 31 01:05:40.706473 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:40.706434    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd" podUID="9b0715eb-599c-4663-874c-b350f7a2910d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:05:40.706473 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:40.706455    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79" podUID="0f28a16d-689f-4baf-96ac-c2c9e7908888" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:05:44.690283 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.690250    4327 generic.go:356] "Generic (PLEG): container finished" podID="9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3" containerID="643e90a2b6044dcd604705694292c07f068e0758c6f11dc689b1e4ef6d9a8604" exitCode=0
Oct 31 01:05:44.690695 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.690285    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz" event={"ID":"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3","Type":"ContainerDied","Data":"643e90a2b6044dcd604705694292c07f068e0758c6f11dc689b1e4ef6d9a8604"}
Oct 31 01:05:44.690695 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.690357    4327 scope.go:122] "RemoveContainer" containerID="36af665e2edb2165b20f033f90248a1eb21445866c3f379f14a0d09c59861dcc"
Oct 31 01:05:44.727332 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.727307    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 01:05:44.792365 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.792334    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-envoy\") pod \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") "
Oct 31 01:05:44.792574 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.792391    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istiod-ca-cert\") pod \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") "
Oct 31 01:05:44.792574 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.792467    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-workload-certs\") pod \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") "
Oct 31 01:05:44.792574 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.792552    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-podinfo\") pod \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") "
Oct 31 01:05:44.792795 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.792582    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-workload-socket\") pod \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") "
Oct 31 01:05:44.792795 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.792750    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-data\") pod \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") "
Oct 31 01:05:44.792918 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.792822    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-workload-certs" pod "9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3" (UID: "9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:05:44.792918 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.792829    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-credential-socket\") pod \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") "
Oct 31 01:05:44.792918 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.792866    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istiod-ca-cert" pod "9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3" (UID: "9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:44.793076 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.792946    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-kube-api-access-sgdzb\" (UniqueName: \"kubernetes.io/projected/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-kube-api-access-sgdzb\") pod \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") "
Oct 31 01:05:44.793076 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.792999    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-credential-socket" pod "9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3" (UID: "9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:05:44.793076 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.793012    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-token\" (UniqueName: \"kubernetes.io/projected/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-token\") pod \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\" (UID: \"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3\") "
Oct 31 01:05:44.793076 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.793016    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-workload-socket" pod "9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3" (UID: "9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:05:44.793279 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.793156    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-data" pod "9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3" (UID: "9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:05:44.793719 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.793696    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:44.793839 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.793721    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:44.793839 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.793738    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:44.793839 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.793751    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:44.793839 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.793765    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:44.794914 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.794890    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-envoy" pod "9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3" (UID: "9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:05:44.795187 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.795164    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-podinfo" pod "9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3" (UID: "9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:05:44.795366 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.795348    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-token" pod "9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3" (UID: "9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:44.795425 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.795406    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-kube-api-access-sgdzb" pod "9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3" (UID: "9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3"). InnerVolumeSpecName "kube-api-access-sgdzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:44.895323 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.895275    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgdzb\" (UniqueName: \"kubernetes.io/projected/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-kube-api-access-sgdzb\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:44.895323 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.895315    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:44.895323 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.895329    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:44.895577 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:44.895340    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:45.294737 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:05:45.294693    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:05:45.698650 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:45.698618    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"
Oct 31 01:05:45.699157 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:45.698660    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz" event={"ID":"9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3","Type":"ContainerDied","Data":"c78eb8c31ad35fbfb7e97b63e388753bb152e14c3fe6a29708cb6c4e75e74265"}
Oct 31 01:05:45.699157 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:45.698739    4327 scope.go:122] "RemoveContainer" containerID="643e90a2b6044dcd604705694292c07f068e0758c6f11dc689b1e4ef6d9a8604"
Oct 31 01:05:45.788148 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:45.788108    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"]
Oct 31 01:05:45.799342 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:45.799308    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-01a117-openshift-default-56c5c46dc5-r9jtz"]
Oct 31 01:05:45.933854 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:45.933805    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3" path="/var/lib/kubelet/pods/9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3/volumes"
Oct 31 01:05:50.719113 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:50.719081    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd"
Oct 31 01:05:50.750960 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:50.750889    4327 generic.go:356] "Generic (PLEG): container finished" podID="0f28a16d-689f-4baf-96ac-c2c9e7908888" containerID="5594c9583f5073f1d392ece714a8d3fe3ee0cafac91459c9942fc30e3d35d4ba" exitCode=0
Oct 31 01:05:50.751102 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:50.750972    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79" event={"ID":"0f28a16d-689f-4baf-96ac-c2c9e7908888","Type":"ContainerDied","Data":"5594c9583f5073f1d392ece714a8d3fe3ee0cafac91459c9942fc30e3d35d4ba"}
Oct 31 01:05:50.752631 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:50.752606    4327 generic.go:356] "Generic (PLEG): container finished" podID="9b0715eb-599c-4663-874c-b350f7a2910d" containerID="add5fa3e5de12e3c452c9c4ed39c39ea47f54b10873a2db002995a724c2428c8" exitCode=0
Oct 31 01:05:50.752751 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:50.752657    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd"
Oct 31 01:05:50.752751 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:50.752667    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd" event={"ID":"9b0715eb-599c-4663-874c-b350f7a2910d","Type":"ContainerDied","Data":"add5fa3e5de12e3c452c9c4ed39c39ea47f54b10873a2db002995a724c2428c8"}
Oct 31 01:05:50.752751 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:50.752743    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd" event={"ID":"9b0715eb-599c-4663-874c-b350f7a2910d","Type":"ContainerDied","Data":"fd328b4298b0b41dcb0fbf16417c62f60d0e8f033e753e8692d3e5980f4ddd93"}
Oct 31 01:05:50.752881 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:50.752767    4327 scope.go:122] "RemoveContainer" containerID="add5fa3e5de12e3c452c9c4ed39c39ea47f54b10873a2db002995a724c2428c8"
Oct 31 01:05:50.773346 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:50.773319    4327 scope.go:122] "RemoveContainer" containerID="add5fa3e5de12e3c452c9c4ed39c39ea47f54b10873a2db002995a724c2428c8"
Oct 31 01:05:50.773645 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:05:50.773613    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"add5fa3e5de12e3c452c9c4ed39c39ea47f54b10873a2db002995a724c2428c8\": container with ID starting with add5fa3e5de12e3c452c9c4ed39c39ea47f54b10873a2db002995a724c2428c8 not found: ID does not exist" containerID="add5fa3e5de12e3c452c9c4ed39c39ea47f54b10873a2db002995a724c2428c8"
Oct 31 01:05:50.773787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:50.773658    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"add5fa3e5de12e3c452c9c4ed39c39ea47f54b10873a2db002995a724c2428c8"} err="failed to get container status \"add5fa3e5de12e3c452c9c4ed39c39ea47f54b10873a2db002995a724c2428c8\": rpc error: code = NotFound desc = could not find container \"add5fa3e5de12e3c452c9c4ed39c39ea47f54b10873a2db002995a724c2428c8\": container with ID starting with add5fa3e5de12e3c452c9c4ed39c39ea47f54b10873a2db002995a724c2428c8 not found: ID does not exist"
Oct 31 01:05:50.776179 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:50.776158    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79"
Oct 31 01:05:50.910098 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:50.910054    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b0715eb-599c-4663-874c-b350f7a2910d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0715eb-599c-4663-874c-b350f7a2910d-plugins-config-volume\") pod \"9b0715eb-599c-4663-874c-b350f7a2910d\" (UID: \"9b0715eb-599c-4663-874c-b350f7a2910d\") "
Oct 31 01:05:50.910301 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:50.910128    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f28a16d-689f-4baf-96ac-c2c9e7908888-kube-api-access-2dfpn\" (UniqueName: \"kubernetes.io/projected/0f28a16d-689f-4baf-96ac-c2c9e7908888-kube-api-access-2dfpn\") pod \"0f28a16d-689f-4baf-96ac-c2c9e7908888\" (UID: \"0f28a16d-689f-4baf-96ac-c2c9e7908888\") "
Oct 31 01:05:50.910301 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:50.910192    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b0715eb-599c-4663-874c-b350f7a2910d-kube-api-access-587rg\" (UniqueName: \"kubernetes.io/projected/9b0715eb-599c-4663-874c-b350f7a2910d-kube-api-access-587rg\") pod \"9b0715eb-599c-4663-874c-b350f7a2910d\" (UID: \"9b0715eb-599c-4663-874c-b350f7a2910d\") "
Oct 31 01:05:50.910408 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:50.910298    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f28a16d-689f-4baf-96ac-c2c9e7908888-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f28a16d-689f-4baf-96ac-c2c9e7908888-plugins-config-volume\") pod \"0f28a16d-689f-4baf-96ac-c2c9e7908888\" (UID: \"0f28a16d-689f-4baf-96ac-c2c9e7908888\") "
Oct 31 01:05:50.911580 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:50.911552    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0715eb-599c-4663-874c-b350f7a2910d-plugins-config-volume" pod "9b0715eb-599c-4663-874c-b350f7a2910d" (UID: "9b0715eb-599c-4663-874c-b350f7a2910d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:50.921713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:50.914374    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f28a16d-689f-4baf-96ac-c2c9e7908888-kube-api-access-2dfpn" pod "0f28a16d-689f-4baf-96ac-c2c9e7908888" (UID: "0f28a16d-689f-4baf-96ac-c2c9e7908888"). InnerVolumeSpecName "kube-api-access-2dfpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:50.921713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:50.917637    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0715eb-599c-4663-874c-b350f7a2910d-kube-api-access-587rg" pod "9b0715eb-599c-4663-874c-b350f7a2910d" (UID: "9b0715eb-599c-4663-874c-b350f7a2910d"). InnerVolumeSpecName "kube-api-access-587rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:50.921713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:50.921322    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f28a16d-689f-4baf-96ac-c2c9e7908888-plugins-config-volume" pod "0f28a16d-689f-4baf-96ac-c2c9e7908888" (UID: "0f28a16d-689f-4baf-96ac-c2c9e7908888"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:51.012625 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:51.012536    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0715eb-599c-4663-874c-b350f7a2910d-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:51.012625 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:51.012569    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dfpn\" (UniqueName: \"kubernetes.io/projected/0f28a16d-689f-4baf-96ac-c2c9e7908888-kube-api-access-2dfpn\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:51.012625 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:51.012579    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-587rg\" (UniqueName: \"kubernetes.io/projected/9b0715eb-599c-4663-874c-b350f7a2910d-kube-api-access-587rg\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:51.012625 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:51.012587    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f28a16d-689f-4baf-96ac-c2c9e7908888-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:05:51.099254 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:51.099218    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd"]
Oct 31 01:05:51.110914 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:51.110883    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd"]
Oct 31 01:05:51.704858 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:51.704816    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79" podUID="0f28a16d-689f-4baf-96ac-c2c9e7908888" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.156:50052\" within 1s: context deadline exceeded"
Oct 31 01:05:51.705048 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:05:51.704874    4327 logging.go:55] [core] [Channel #8703 SubChannel #8704]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.156:50052", ServerName: "10.133.0.156:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.156:50052: operation was canceled"
Oct 31 01:05:51.705048 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:05:51.704872    4327 logging.go:55] [core] [Channel #8701 SubChannel #8702]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.157:50052", ServerName: "10.133.0.157:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.157:50052: operation was canceled"
Oct 31 01:05:51.705048 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:51.704814    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-72120c-5d4968bd99-fz4dd" podUID="9b0715eb-599c-4663-874c-b350f7a2910d" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.157:50052\" within 1s: context deadline exceeded"
Oct 31 01:05:51.705048 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:51.704901    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79"
Oct 31 01:05:51.761268 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:51.761233    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79"
Oct 31 01:05:51.761756 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:51.761275    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79" event={"ID":"0f28a16d-689f-4baf-96ac-c2c9e7908888","Type":"ContainerDied","Data":"0e0abcfaa66d3910d089da45871f7fbc0d628f4a2121e1bc7b08d074a80edb35"}
Oct 31 01:05:51.761756 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:51.761347    4327 scope.go:122] "RemoveContainer" containerID="5594c9583f5073f1d392ece714a8d3fe3ee0cafac91459c9942fc30e3d35d4ba"
Oct 31 01:05:51.815251 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:51.815217    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79"]
Oct 31 01:05:51.830997 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:51.830967    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72120c-7776cf45f-xlr79"]
Oct 31 01:05:51.922330 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:51.922297    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f28a16d-689f-4baf-96ac-c2c9e7908888" path="/var/lib/kubelet/pods/0f28a16d-689f-4baf-96ac-c2c9e7908888/volumes"
Oct 31 01:05:51.922982 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:51.922960    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b0715eb-599c-4663-874c-b350f7a2910d" path="/var/lib/kubelet/pods/9b0715eb-599c-4663-874c-b350f7a2910d/volumes"
Oct 31 01:05:55.111646 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.111610    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"]
Oct 31 01:05:55.112365 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.112349    4327 cpu_manager.go:414] "containerMap: removing container" podUID="9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3" containerName="istio-proxy"
Oct 31 01:05:55.112410 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.112366    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3" containerName="istio-proxy"
Oct 31 01:05:55.112410 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.112382    4327 cpu_manager.go:414] "containerMap: removing container" podUID="cf355564-5ea9-4149-9540-d01964674903" containerName="payload-processing"
Oct 31 01:05:55.112410 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.112387    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf355564-5ea9-4149-9540-d01964674903" containerName="payload-processing"
Oct 31 01:05:55.112410 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.112397    4327 cpu_manager.go:414] "containerMap: removing container" podUID="9b0715eb-599c-4663-874c-b350f7a2910d" containerName="payload-processing"
Oct 31 01:05:55.112410 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.112404    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b0715eb-599c-4663-874c-b350f7a2910d" containerName="payload-processing"
Oct 31 01:05:55.112410 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.112412    4327 cpu_manager.go:414] "containerMap: removing container" podUID="e0a4bc42-fcaa-47bc-ad35-a654443342c1" containerName="main"
Oct 31 01:05:55.112602 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.112417    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0a4bc42-fcaa-47bc-ad35-a654443342c1" containerName="main"
Oct 31 01:05:55.112602 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.112431    4327 cpu_manager.go:414] "containerMap: removing container" podUID="0f28a16d-689f-4baf-96ac-c2c9e7908888" containerName="payload-pre-processing"
Oct 31 01:05:55.112602 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.112436    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f28a16d-689f-4baf-96ac-c2c9e7908888" containerName="payload-pre-processing"
Oct 31 01:05:55.112602 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.112451    4327 cpu_manager.go:414] "containerMap: removing container" podUID="9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3" containerName="istio-proxy"
Oct 31 01:05:55.112602 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.112456    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3" containerName="istio-proxy"
Oct 31 01:05:55.112602 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.112482    4327 cpu_manager.go:414] "containerMap: removing container" podUID="57e3b4e5-4f46-4c99-a260-3beb556431c7" containerName="payload-pre-processing"
Oct 31 01:05:55.112602 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.112486    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57e3b4e5-4f46-4c99-a260-3beb556431c7" containerName="payload-pre-processing"
Oct 31 01:05:55.112602 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.112598    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="57e3b4e5-4f46-4c99-a260-3beb556431c7" containerName="payload-pre-processing"
Oct 31 01:05:55.112852 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.112607    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3" containerName="istio-proxy"
Oct 31 01:05:55.112852 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.112616    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf355564-5ea9-4149-9540-d01964674903" containerName="payload-processing"
Oct 31 01:05:55.112852 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.112621    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b0715eb-599c-4663-874c-b350f7a2910d" containerName="payload-processing"
Oct 31 01:05:55.112852 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.112626    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0a4bc42-fcaa-47bc-ad35-a654443342c1" containerName="main"
Oct 31 01:05:55.112852 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.112635    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f28a16d-689f-4baf-96ac-c2c9e7908888" containerName="payload-pre-processing"
Oct 31 01:05:55.112852 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.112642    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a7a3f22-3f9e-46a4-bed7-5591bdbcb3f3" containerName="istio-proxy"
Oct 31 01:05:55.117618 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.117602    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"
Oct 31 01:05:55.155038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.154997    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-36af31\"/\"openshift-service-ca.crt\""
Oct 31 01:05:55.155419 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.155397    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-36af31\"/\"default-dockercfg-hz9tr\""
Oct 31 01:05:55.155607 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.155526    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-36af31\"/\"kube-root-ca.crt\""
Oct 31 01:05:55.155752 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.155610    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-36af31\"/\"auth-test-model-36af31-kserve-self-signed-certs\""
Oct 31 01:05:55.161342 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.161311    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"]
Oct 31 01:05:55.185964 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.185926    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsd22\" (UniqueName: \"kubernetes.io/projected/ef6850b2-d382-4728-9f01-72e9639adefc-kube-api-access-xsd22\") pod \"auth-test-model-36af31-kserve-847b468c57-bxs2w\" (UID: \"ef6850b2-d382-4728-9f01-72e9639adefc\") " pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"
Oct 31 01:05:55.186132 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.185988    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-model-cache\") pod \"auth-test-model-36af31-kserve-847b468c57-bxs2w\" (UID: \"ef6850b2-d382-4728-9f01-72e9639adefc\") " pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"
Oct 31 01:05:55.186181 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.186138    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-tmp-dir\") pod \"auth-test-model-36af31-kserve-847b468c57-bxs2w\" (UID: \"ef6850b2-d382-4728-9f01-72e9639adefc\") " pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"
Oct 31 01:05:55.186316 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.186287    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-dshm\") pod \"auth-test-model-36af31-kserve-847b468c57-bxs2w\" (UID: \"ef6850b2-d382-4728-9f01-72e9639adefc\") " pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"
Oct 31 01:05:55.186369 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.186337    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-home\") pod \"auth-test-model-36af31-kserve-847b468c57-bxs2w\" (UID: \"ef6850b2-d382-4728-9f01-72e9639adefc\") " pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"
Oct 31 01:05:55.186467 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.186447    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef6850b2-d382-4728-9f01-72e9639adefc-tls-certs\") pod \"auth-test-model-36af31-kserve-847b468c57-bxs2w\" (UID: \"ef6850b2-d382-4728-9f01-72e9639adefc\") " pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"
Oct 31 01:05:55.289728 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.289187    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsd22\" (UniqueName: \"kubernetes.io/projected/ef6850b2-d382-4728-9f01-72e9639adefc-kube-api-access-xsd22\") pod \"auth-test-model-36af31-kserve-847b468c57-bxs2w\" (UID: \"ef6850b2-d382-4728-9f01-72e9639adefc\") " pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"
Oct 31 01:05:55.289937 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.289851    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-model-cache\") pod \"auth-test-model-36af31-kserve-847b468c57-bxs2w\" (UID: \"ef6850b2-d382-4728-9f01-72e9639adefc\") " pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"
Oct 31 01:05:55.290003 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.289990    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-tmp-dir\") pod \"auth-test-model-36af31-kserve-847b468c57-bxs2w\" (UID: \"ef6850b2-d382-4728-9f01-72e9639adefc\") " pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"
Oct 31 01:05:55.290137 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.290120    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-dshm\") pod \"auth-test-model-36af31-kserve-847b468c57-bxs2w\" (UID: \"ef6850b2-d382-4728-9f01-72e9639adefc\") " pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"
Oct 31 01:05:55.290210 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.290156    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-home\") pod \"auth-test-model-36af31-kserve-847b468c57-bxs2w\" (UID: \"ef6850b2-d382-4728-9f01-72e9639adefc\") " pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"
Oct 31 01:05:55.290276 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.290223    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef6850b2-d382-4728-9f01-72e9639adefc-tls-certs\") pod \"auth-test-model-36af31-kserve-847b468c57-bxs2w\" (UID: \"ef6850b2-d382-4728-9f01-72e9639adefc\") " pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"
Oct 31 01:05:55.290363 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.290341    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-model-cache\") pod \"auth-test-model-36af31-kserve-847b468c57-bxs2w\" (UID: \"ef6850b2-d382-4728-9f01-72e9639adefc\") " pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"
Oct 31 01:05:55.290489 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.290472    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-tmp-dir\") pod \"auth-test-model-36af31-kserve-847b468c57-bxs2w\" (UID: \"ef6850b2-d382-4728-9f01-72e9639adefc\") " pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"
Oct 31 01:05:55.290599 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.290568    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-home\") pod \"auth-test-model-36af31-kserve-847b468c57-bxs2w\" (UID: \"ef6850b2-d382-4728-9f01-72e9639adefc\") " pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"
Oct 31 01:05:55.302314 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.292842    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-dshm\") pod \"auth-test-model-36af31-kserve-847b468c57-bxs2w\" (UID: \"ef6850b2-d382-4728-9f01-72e9639adefc\") " pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"
Oct 31 01:05:55.302314 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.293205    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef6850b2-d382-4728-9f01-72e9639adefc-tls-certs\") pod \"auth-test-model-36af31-kserve-847b468c57-bxs2w\" (UID: \"ef6850b2-d382-4728-9f01-72e9639adefc\") " pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"
Oct 31 01:05:55.341714 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.334554    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsd22\" (UniqueName: \"kubernetes.io/projected/ef6850b2-d382-4728-9f01-72e9639adefc-kube-api-access-xsd22\") pod \"auth-test-model-36af31-kserve-847b468c57-bxs2w\" (UID: \"ef6850b2-d382-4728-9f01-72e9639adefc\") " pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"
Oct 31 01:05:55.433760 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.431997    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"
Oct 31 01:05:55.746251 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:05:55.746223    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef6850b2_d382_4728_9f01_72e9639adefc.slice/crio-37e05e63f9997a514b861bd8fc6ee767c092fdab63fcaa5e6045edf674a3bfce WatchSource:0}: Error finding container 37e05e63f9997a514b861bd8fc6ee767c092fdab63fcaa5e6045edf674a3bfce: Status 404 returned error can't find the container with id 37e05e63f9997a514b861bd8fc6ee767c092fdab63fcaa5e6045edf674a3bfce
Oct 31 01:05:55.746803 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.746762    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"]
Oct 31 01:05:55.797242 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:55.797206    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w" event={"ID":"ef6850b2-d382-4728-9f01-72e9639adefc","Type":"ContainerStarted","Data":"37e05e63f9997a514b861bd8fc6ee767c092fdab63fcaa5e6045edf674a3bfce"}
Oct 31 01:05:56.806976 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:56.806901    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w" event={"ID":"ef6850b2-d382-4728-9f01-72e9639adefc","Type":"ContainerStarted","Data":"9512722b45aa5f454a7bfbf95c7664cb5c24fc729b9c6d0464bcd423b50705bb"}
Oct 31 01:05:56.852555 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:56.852499    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w" podStartSLOduration=1.85248348 podStartE2EDuration="1.85248348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:05:56.848311196 +0000 UTC m=+2817.070346740" watchObservedRunningTime="2026-10-31 01:05:56.85248348 +0000 UTC m=+2817.074519022"
Oct 31 01:05:57.303974 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:57.303938    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng"]
Oct 31 01:05:57.365072 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:57.365034    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8"]
Oct 31 01:05:57.432913 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:05:57.432883    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"
Oct 31 01:06:02.305045 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:02.304998    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng" podUID="2453aadb-8cf3-403f-b728-d4de97020398" containerName="payload-processing" containerID="cri-o://7cd05e2f48cae0f1839af8c672c83faa72cfb02f20a402ee124adab5523f43e7" gracePeriod=25
Oct 31 01:06:02.365890 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:02.365770    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8" podUID="fd9b6201-5de2-4147-94c2-edd87d1ec204" containerName="payload-pre-processing" containerID="cri-o://8cb7195d654b8bf72ab1e1297e4ae887c8ecfa2cd5cdab1a7e9ed75c25aec6db" gracePeriod=25
Oct 31 01:06:02.579894 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:02.579869    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng"
Oct 31 01:06:02.708537 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:02.708499    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2453aadb-8cf3-403f-b728-d4de97020398-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2453aadb-8cf3-403f-b728-d4de97020398-plugins-config-volume\") pod \"2453aadb-8cf3-403f-b728-d4de97020398\" (UID: \"2453aadb-8cf3-403f-b728-d4de97020398\") "
Oct 31 01:06:02.708771 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:02.708754    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2453aadb-8cf3-403f-b728-d4de97020398-kube-api-access-pwt9d\" (UniqueName: \"kubernetes.io/projected/2453aadb-8cf3-403f-b728-d4de97020398-kube-api-access-pwt9d\") pod \"2453aadb-8cf3-403f-b728-d4de97020398\" (UID: \"2453aadb-8cf3-403f-b728-d4de97020398\") "
Oct 31 01:06:02.708936 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:02.708911    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2453aadb-8cf3-403f-b728-d4de97020398-plugins-config-volume" pod "2453aadb-8cf3-403f-b728-d4de97020398" (UID: "2453aadb-8cf3-403f-b728-d4de97020398"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:06:02.709200 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:02.709185    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2453aadb-8cf3-403f-b728-d4de97020398-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:06:02.711030 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:02.711010    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2453aadb-8cf3-403f-b728-d4de97020398-kube-api-access-pwt9d" pod "2453aadb-8cf3-403f-b728-d4de97020398" (UID: "2453aadb-8cf3-403f-b728-d4de97020398"). InnerVolumeSpecName "kube-api-access-pwt9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:06:02.810395 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:02.810302    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwt9d\" (UniqueName: \"kubernetes.io/projected/2453aadb-8cf3-403f-b728-d4de97020398-kube-api-access-pwt9d\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:06:02.852118 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:02.852084    4327 generic.go:356] "Generic (PLEG): container finished" podID="2453aadb-8cf3-403f-b728-d4de97020398" containerID="7cd05e2f48cae0f1839af8c672c83faa72cfb02f20a402ee124adab5523f43e7" exitCode=0
Oct 31 01:06:02.852304 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:02.852137    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng"
Oct 31 01:06:02.852304 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:02.852166    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng" event={"ID":"2453aadb-8cf3-403f-b728-d4de97020398","Type":"ContainerDied","Data":"7cd05e2f48cae0f1839af8c672c83faa72cfb02f20a402ee124adab5523f43e7"}
Oct 31 01:06:02.852304 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:02.852229    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng" event={"ID":"2453aadb-8cf3-403f-b728-d4de97020398","Type":"ContainerDied","Data":"715c753c24be6251a9b6dc5991d3766fe7e2d4962f9619a7fb547eb976f58fb8"}
Oct 31 01:06:02.852304 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:02.852253    4327 scope.go:122] "RemoveContainer" containerID="7cd05e2f48cae0f1839af8c672c83faa72cfb02f20a402ee124adab5523f43e7"
Oct 31 01:06:02.864531 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:02.864511    4327 scope.go:122] "RemoveContainer" containerID="7cd05e2f48cae0f1839af8c672c83faa72cfb02f20a402ee124adab5523f43e7"
Oct 31 01:06:02.864863 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:06:02.864840    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cd05e2f48cae0f1839af8c672c83faa72cfb02f20a402ee124adab5523f43e7\": container with ID starting with 7cd05e2f48cae0f1839af8c672c83faa72cfb02f20a402ee124adab5523f43e7 not found: ID does not exist" containerID="7cd05e2f48cae0f1839af8c672c83faa72cfb02f20a402ee124adab5523f43e7"
Oct 31 01:06:02.864945 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:02.864870    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd05e2f48cae0f1839af8c672c83faa72cfb02f20a402ee124adab5523f43e7"} err="failed to get container status \"7cd05e2f48cae0f1839af8c672c83faa72cfb02f20a402ee124adab5523f43e7\": rpc error: code = NotFound desc = could not find container \"7cd05e2f48cae0f1839af8c672c83faa72cfb02f20a402ee124adab5523f43e7\": container with ID starting with 7cd05e2f48cae0f1839af8c672c83faa72cfb02f20a402ee124adab5523f43e7 not found: ID does not exist"
Oct 31 01:06:02.909503 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:02.909471    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng"]
Oct 31 01:06:02.921554 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:02.921523    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-82b5e8-5cc6c765cf-4z7ng"]
Oct 31 01:06:03.394475 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:03.394429    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8" podUID="fd9b6201-5de2-4147-94c2-edd87d1ec204" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:06:03.922763 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:03.922729    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2453aadb-8cf3-403f-b728-d4de97020398" path="/var/lib/kubelet/pods/2453aadb-8cf3-403f-b728-d4de97020398/volumes"
Oct 31 01:06:05.433263 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:05.433225    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"
Oct 31 01:06:05.446813 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:05.446788    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"
Oct 31 01:06:05.884853 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:05.884820    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"
Oct 31 01:06:13.394983 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:13.394884    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8" podUID="fd9b6201-5de2-4147-94c2-edd87d1ec204" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:06:15.292921 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:06:15.292881    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:06:22.663778 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:22.663747    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8"
Oct 31 01:06:22.805622 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:22.805591    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd9b6201-5de2-4147-94c2-edd87d1ec204-kube-api-access-w5r5q\" (UniqueName: \"kubernetes.io/projected/fd9b6201-5de2-4147-94c2-edd87d1ec204-kube-api-access-w5r5q\") pod \"fd9b6201-5de2-4147-94c2-edd87d1ec204\" (UID: \"fd9b6201-5de2-4147-94c2-edd87d1ec204\") "
Oct 31 01:06:22.805856 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:22.805744    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd9b6201-5de2-4147-94c2-edd87d1ec204-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd9b6201-5de2-4147-94c2-edd87d1ec204-plugins-config-volume\") pod \"fd9b6201-5de2-4147-94c2-edd87d1ec204\" (UID: \"fd9b6201-5de2-4147-94c2-edd87d1ec204\") "
Oct 31 01:06:22.806121 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:22.806085    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd9b6201-5de2-4147-94c2-edd87d1ec204-plugins-config-volume" pod "fd9b6201-5de2-4147-94c2-edd87d1ec204" (UID: "fd9b6201-5de2-4147-94c2-edd87d1ec204"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:06:22.806542 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:22.806516    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd9b6201-5de2-4147-94c2-edd87d1ec204-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:06:22.808123 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:22.808095    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9b6201-5de2-4147-94c2-edd87d1ec204-kube-api-access-w5r5q" pod "fd9b6201-5de2-4147-94c2-edd87d1ec204" (UID: "fd9b6201-5de2-4147-94c2-edd87d1ec204"). InnerVolumeSpecName "kube-api-access-w5r5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:06:22.907907 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:22.907819    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5r5q\" (UniqueName: \"kubernetes.io/projected/fd9b6201-5de2-4147-94c2-edd87d1ec204-kube-api-access-w5r5q\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:06:23.013141 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:23.013096    4327 generic.go:356] "Generic (PLEG): container finished" podID="fd9b6201-5de2-4147-94c2-edd87d1ec204" containerID="8cb7195d654b8bf72ab1e1297e4ae887c8ecfa2cd5cdab1a7e9ed75c25aec6db" exitCode=0
Oct 31 01:06:23.013141 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:23.013142    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8"
Oct 31 01:06:23.013456 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:23.013159    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8" event={"ID":"fd9b6201-5de2-4147-94c2-edd87d1ec204","Type":"ContainerDied","Data":"8cb7195d654b8bf72ab1e1297e4ae887c8ecfa2cd5cdab1a7e9ed75c25aec6db"}
Oct 31 01:06:23.013456 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:23.013214    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8" event={"ID":"fd9b6201-5de2-4147-94c2-edd87d1ec204","Type":"ContainerDied","Data":"10f8da7945a736c59414fddc2d925dd53b9de0bef92499022437ba526dbf790c"}
Oct 31 01:06:23.013456 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:23.013240    4327 scope.go:122] "RemoveContainer" containerID="8cb7195d654b8bf72ab1e1297e4ae887c8ecfa2cd5cdab1a7e9ed75c25aec6db"
Oct 31 01:06:23.025183 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:23.025160    4327 scope.go:122] "RemoveContainer" containerID="8cb7195d654b8bf72ab1e1297e4ae887c8ecfa2cd5cdab1a7e9ed75c25aec6db"
Oct 31 01:06:23.025567 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:06:23.025541    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cb7195d654b8bf72ab1e1297e4ae887c8ecfa2cd5cdab1a7e9ed75c25aec6db\": container with ID starting with 8cb7195d654b8bf72ab1e1297e4ae887c8ecfa2cd5cdab1a7e9ed75c25aec6db not found: ID does not exist" containerID="8cb7195d654b8bf72ab1e1297e4ae887c8ecfa2cd5cdab1a7e9ed75c25aec6db"
Oct 31 01:06:23.025699 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:23.025574    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cb7195d654b8bf72ab1e1297e4ae887c8ecfa2cd5cdab1a7e9ed75c25aec6db"} err="failed to get container status \"8cb7195d654b8bf72ab1e1297e4ae887c8ecfa2cd5cdab1a7e9ed75c25aec6db\": rpc error: code = NotFound desc = could not find container \"8cb7195d654b8bf72ab1e1297e4ae887c8ecfa2cd5cdab1a7e9ed75c25aec6db\": container with ID starting with 8cb7195d654b8bf72ab1e1297e4ae887c8ecfa2cd5cdab1a7e9ed75c25aec6db not found: ID does not exist"
Oct 31 01:06:23.067114 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:23.067077    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8"]
Oct 31 01:06:23.076313 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:23.076281    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-82b5e8-77c7fb89bc-2cvn8"]
Oct 31 01:06:23.939491 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:23.939449    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd9b6201-5de2-4147-94c2-edd87d1ec204" path="/var/lib/kubelet/pods/fd9b6201-5de2-4147-94c2-edd87d1ec204/volumes"
Oct 31 01:06:36.490373 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:36.490332    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"]
Oct 31 01:06:36.490979 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:36.490543    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65" podUID="7a206da9-cfc0-41d4-888c-1123aab28e5c" containerName="istio-proxy" containerID="cri-o://4f90bb472df3a4a3cb5967b8be094dfa52ae382eddd6cc82b4b2f4d108f910ab" gracePeriod=30
Oct 31 01:06:40.469526 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.469484    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"]
Oct 31 01:06:40.470265 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.470247    4327 cpu_manager.go:414] "containerMap: removing container" podUID="2453aadb-8cf3-403f-b728-d4de97020398" containerName="payload-processing"
Oct 31 01:06:40.470265 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.470267    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2453aadb-8cf3-403f-b728-d4de97020398" containerName="payload-processing"
Oct 31 01:06:40.470368 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.470306    4327 cpu_manager.go:414] "containerMap: removing container" podUID="fd9b6201-5de2-4147-94c2-edd87d1ec204" containerName="payload-pre-processing"
Oct 31 01:06:40.470368 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.470312    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd9b6201-5de2-4147-94c2-edd87d1ec204" containerName="payload-pre-processing"
Oct 31 01:06:40.470467 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.470417    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd9b6201-5de2-4147-94c2-edd87d1ec204" containerName="payload-pre-processing"
Oct 31 01:06:40.470467 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.470430    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="2453aadb-8cf3-403f-b728-d4de97020398" containerName="payload-processing"
Oct 31 01:06:40.474189 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.474168    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"
Oct 31 01:06:40.497498 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.497464    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"]
Oct 31 01:06:40.501639 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.501609    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-home\") pod \"auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w\" (UID: \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"
Oct 31 01:06:40.501827 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.501649    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-model-cache\") pod \"auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w\" (UID: \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"
Oct 31 01:06:40.501827 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.501790    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-tls-certs\") pod \"auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w\" (UID: \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"
Oct 31 01:06:40.501964 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.501830    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-dshm\") pod \"auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w\" (UID: \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"
Oct 31 01:06:40.501964 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.501941    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q6nc\" (UniqueName: \"kubernetes.io/projected/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-kube-api-access-8q6nc\") pod \"auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w\" (UID: \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"
Oct 31 01:06:40.502119 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.502097    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-tmp-dir\") pod \"auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w\" (UID: \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"
Oct 31 01:06:40.503398 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.503370    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-f2bdf3\"/\"openshift-service-ca.crt\""
Oct 31 01:06:40.503524 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.503467    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-f2bdf3\"/\"kube-root-ca.crt\""
Oct 31 01:06:40.506117 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.506095    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-f2bdf3\"/\"auth-test-model-f2bdf3-kserve-self-signed-certs\""
Oct 31 01:06:40.506117 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.506105    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-f2bdf3\"/\"default-dockercfg-w6mb7\""
Oct 31 01:06:40.603153 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.603116    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-tmp-dir\") pod \"auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w\" (UID: \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"
Oct 31 01:06:40.603337 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.603180    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-home\") pod \"auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w\" (UID: \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"
Oct 31 01:06:40.603337 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.603205    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-model-cache\") pod \"auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w\" (UID: \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"
Oct 31 01:06:40.603337 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.603276    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-tls-certs\") pod \"auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w\" (UID: \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"
Oct 31 01:06:40.603337 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.603310    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-dshm\") pod \"auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w\" (UID: \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"
Oct 31 01:06:40.603523 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.603407    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q6nc\" (UniqueName: \"kubernetes.io/projected/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-kube-api-access-8q6nc\") pod \"auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w\" (UID: \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"
Oct 31 01:06:40.603604 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.603581    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-tmp-dir\") pod \"auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w\" (UID: \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"
Oct 31 01:06:40.603710 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.603599    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-model-cache\") pod \"auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w\" (UID: \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"
Oct 31 01:06:40.603710 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.603658    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-home\") pod \"auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w\" (UID: \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"
Oct 31 01:06:40.605601 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.605572    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-dshm\") pod \"auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w\" (UID: \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"
Oct 31 01:06:40.605838 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.605821    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-tls-certs\") pod \"auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w\" (UID: \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"
Oct 31 01:06:40.627266 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.627233    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q6nc\" (UniqueName: \"kubernetes.io/projected/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-kube-api-access-8q6nc\") pod \"auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w\" (UID: \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"
Oct 31 01:06:40.803754 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:40.794985    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"
Oct 31 01:06:41.154381 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:06:41.154351    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cc1b090_45a2_4aa0_a778_a4b18f8e197c.slice/crio-171eecb00176b01acd42e5a3a6cfecc02bdb26219bc6486514263b551a9ec78c WatchSource:0}: Error finding container 171eecb00176b01acd42e5a3a6cfecc02bdb26219bc6486514263b551a9ec78c: Status 404 returned error can't find the container with id 171eecb00176b01acd42e5a3a6cfecc02bdb26219bc6486514263b551a9ec78c
Oct 31 01:06:41.155976 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.155939    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"]
Oct 31 01:06:41.184754 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.184725    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w" event={"ID":"9cc1b090-45a2-4aa0-a778-a4b18f8e197c","Type":"ContainerStarted","Data":"171eecb00176b01acd42e5a3a6cfecc02bdb26219bc6486514263b551a9ec78c"}
Oct 31 01:06:41.786653 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.786626    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:06:41.823512 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.823478    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-envoy\") pod \"7a206da9-cfc0-41d4-888c-1123aab28e5c\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") "
Oct 31 01:06:41.823742 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.823594    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-workload-certs\") pod \"7a206da9-cfc0-41d4-888c-1123aab28e5c\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") "
Oct 31 01:06:41.823742 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.823659    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-data\") pod \"7a206da9-cfc0-41d4-888c-1123aab28e5c\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") "
Oct 31 01:06:41.823742 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.823727    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a206da9-cfc0-41d4-888c-1123aab28e5c-kube-api-access-9b2qw\" (UniqueName: \"kubernetes.io/projected/7a206da9-cfc0-41d4-888c-1123aab28e5c-kube-api-access-9b2qw\") pod \"7a206da9-cfc0-41d4-888c-1123aab28e5c\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") "
Oct 31 01:06:41.823939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.823790    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-podinfo\") pod \"7a206da9-cfc0-41d4-888c-1123aab28e5c\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") "
Oct 31 01:06:41.823939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.823816    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-credential-socket\") pod \"7a206da9-cfc0-41d4-888c-1123aab28e5c\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") "
Oct 31 01:06:41.824060 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.823929    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a206da9-cfc0-41d4-888c-1123aab28e5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a206da9-cfc0-41d4-888c-1123aab28e5c-istiod-ca-cert\") pod \"7a206da9-cfc0-41d4-888c-1123aab28e5c\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") "
Oct 31 01:06:41.824060 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.823947    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-data" pod "7a206da9-cfc0-41d4-888c-1123aab28e5c" (UID: "7a206da9-cfc0-41d4-888c-1123aab28e5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:06:41.824060 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.823966    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-workload-certs" pod "7a206da9-cfc0-41d4-888c-1123aab28e5c" (UID: "7a206da9-cfc0-41d4-888c-1123aab28e5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:06:41.824060 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.824025    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-workload-socket\") pod \"7a206da9-cfc0-41d4-888c-1123aab28e5c\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") "
Oct 31 01:06:41.824271 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.824066    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-token\" (UniqueName: \"kubernetes.io/projected/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-token\") pod \"7a206da9-cfc0-41d4-888c-1123aab28e5c\" (UID: \"7a206da9-cfc0-41d4-888c-1123aab28e5c\") "
Oct 31 01:06:41.824271 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.824138    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-credential-socket" pod "7a206da9-cfc0-41d4-888c-1123aab28e5c" (UID: "7a206da9-cfc0-41d4-888c-1123aab28e5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:06:41.824271 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.824260    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a206da9-cfc0-41d4-888c-1123aab28e5c-istiod-ca-cert" pod "7a206da9-cfc0-41d4-888c-1123aab28e5c" (UID: "7a206da9-cfc0-41d4-888c-1123aab28e5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:06:41.824445 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.824369    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-workload-socket" pod "7a206da9-cfc0-41d4-888c-1123aab28e5c" (UID: "7a206da9-cfc0-41d4-888c-1123aab28e5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:06:41.824851 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.824827    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:06:41.824975 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.824856    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a206da9-cfc0-41d4-888c-1123aab28e5c-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:06:41.824975 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.824874    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:06:41.824975 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.824890    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:06:41.824975 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.824904    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:06:41.825937 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.825913    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-envoy" pod "7a206da9-cfc0-41d4-888c-1123aab28e5c" (UID: "7a206da9-cfc0-41d4-888c-1123aab28e5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:06:41.826101 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.826081    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-podinfo" pod "7a206da9-cfc0-41d4-888c-1123aab28e5c" (UID: "7a206da9-cfc0-41d4-888c-1123aab28e5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:06:41.826240 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.826223    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a206da9-cfc0-41d4-888c-1123aab28e5c-kube-api-access-9b2qw" pod "7a206da9-cfc0-41d4-888c-1123aab28e5c" (UID: "7a206da9-cfc0-41d4-888c-1123aab28e5c"). InnerVolumeSpecName "kube-api-access-9b2qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:06:41.826284 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.826248    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-token" pod "7a206da9-cfc0-41d4-888c-1123aab28e5c" (UID: "7a206da9-cfc0-41d4-888c-1123aab28e5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:06:41.926128 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.926096    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:06:41.926128 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.926126    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:06:41.926337 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.926141    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a206da9-cfc0-41d4-888c-1123aab28e5c-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:06:41.926337 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:41.926152    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b2qw\" (UniqueName: \"kubernetes.io/projected/7a206da9-cfc0-41d4-888c-1123aab28e5c-kube-api-access-9b2qw\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:06:42.221786 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:42.221743    4327 generic.go:356] "Generic (PLEG): container finished" podID="7a206da9-cfc0-41d4-888c-1123aab28e5c" containerID="4f90bb472df3a4a3cb5967b8be094dfa52ae382eddd6cc82b4b2f4d108f910ab" exitCode=0
Oct 31 01:06:42.221980 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:42.221840    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"
Oct 31 01:06:42.221980 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:42.221837    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65" event={"ID":"7a206da9-cfc0-41d4-888c-1123aab28e5c","Type":"ContainerDied","Data":"4f90bb472df3a4a3cb5967b8be094dfa52ae382eddd6cc82b4b2f4d108f910ab"}
Oct 31 01:06:42.222130 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:42.222032    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65" event={"ID":"7a206da9-cfc0-41d4-888c-1123aab28e5c","Type":"ContainerDied","Data":"6fd36b749f73c18ed613cc186e341ebf61c6e12edf8ea34c2335cc053b79dce7"}
Oct 31 01:06:42.222130 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:42.222064    4327 scope.go:122] "RemoveContainer" containerID="4f90bb472df3a4a3cb5967b8be094dfa52ae382eddd6cc82b4b2f4d108f910ab"
Oct 31 01:06:42.224599 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:42.224567    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w" event={"ID":"9cc1b090-45a2-4aa0-a778-a4b18f8e197c","Type":"ContainerStarted","Data":"777cfaf996e8915907fe2fbe52676816d1135b132c6dcc17e72a72eae4c8596c"}
Oct 31 01:06:42.234201 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:42.234181    4327 scope.go:122] "RemoveContainer" containerID="4f90bb472df3a4a3cb5967b8be094dfa52ae382eddd6cc82b4b2f4d108f910ab"
Oct 31 01:06:42.234492 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:06:42.234471    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f90bb472df3a4a3cb5967b8be094dfa52ae382eddd6cc82b4b2f4d108f910ab\": container with ID starting with 4f90bb472df3a4a3cb5967b8be094dfa52ae382eddd6cc82b4b2f4d108f910ab not found: ID does not exist" containerID="4f90bb472df3a4a3cb5967b8be094dfa52ae382eddd6cc82b4b2f4d108f910ab"
Oct 31 01:06:42.234564 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:42.234506    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f90bb472df3a4a3cb5967b8be094dfa52ae382eddd6cc82b4b2f4d108f910ab"} err="failed to get container status \"4f90bb472df3a4a3cb5967b8be094dfa52ae382eddd6cc82b4b2f4d108f910ab\": rpc error: code = NotFound desc = could not find container \"4f90bb472df3a4a3cb5967b8be094dfa52ae382eddd6cc82b4b2f4d108f910ab\": container with ID starting with 4f90bb472df3a4a3cb5967b8be094dfa52ae382eddd6cc82b4b2f4d108f910ab not found: ID does not exist"
Oct 31 01:06:42.283180 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:42.283125    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w" podStartSLOduration=2.283110019 podStartE2EDuration="2.283110019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:42.280347387 +0000 UTC m=+2862.502382930" watchObservedRunningTime="2026-10-31 01:06:42.283110019 +0000 UTC m=+2862.505145561"
Oct 31 01:06:42.320257 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:42.320224    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"]
Oct 31 01:06:42.332193 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:42.332157    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-4wp65"]
Oct 31 01:06:42.795802 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:42.795765    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"
Oct 31 01:06:43.180513 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.180475    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x"]
Oct 31 01:06:43.181336 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.181313    4327 cpu_manager.go:414] "containerMap: removing container" podUID="7a206da9-cfc0-41d4-888c-1123aab28e5c" containerName="istio-proxy"
Oct 31 01:06:43.181471 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.181336    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a206da9-cfc0-41d4-888c-1123aab28e5c" containerName="istio-proxy"
Oct 31 01:06:43.181563 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.181549    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a206da9-cfc0-41d4-888c-1123aab28e5c" containerName="istio-proxy"
Oct 31 01:06:43.186741 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.186719    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x"
Oct 31 01:06:43.205020 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.204980    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-d10721\""
Oct 31 01:06:43.205226 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.205043    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-d10721-dockercfg-khsv8\""
Oct 31 01:06:43.206022 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.205997    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x"]
Oct 31 01:06:43.255815 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.247450    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b835588e-1042-46bb-8554-243c7dab259d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x\" (UID: \"b835588e-1042-46bb-8554-243c7dab259d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x"
Oct 31 01:06:43.256361 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.256328    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf6px\" (UniqueName: \"kubernetes.io/projected/b835588e-1042-46bb-8554-243c7dab259d-kube-api-access-kf6px\") pod \"payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x\" (UID: \"b835588e-1042-46bb-8554-243c7dab259d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x"
Oct 31 01:06:43.368223 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.367201    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b835588e-1042-46bb-8554-243c7dab259d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x\" (UID: \"b835588e-1042-46bb-8554-243c7dab259d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x"
Oct 31 01:06:43.368223 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.367369    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf6px\" (UniqueName: \"kubernetes.io/projected/b835588e-1042-46bb-8554-243c7dab259d-kube-api-access-kf6px\") pod \"payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x\" (UID: \"b835588e-1042-46bb-8554-243c7dab259d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x"
Oct 31 01:06:43.368958 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.368924    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b835588e-1042-46bb-8554-243c7dab259d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x\" (UID: \"b835588e-1042-46bb-8554-243c7dab259d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x"
Oct 31 01:06:43.371580 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.371554    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl"]
Oct 31 01:06:43.391223 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.381797    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl"]
Oct 31 01:06:43.391223 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.381910    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl"
Oct 31 01:06:43.426309 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.426275    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf6px\" (UniqueName: \"kubernetes.io/projected/b835588e-1042-46bb-8554-243c7dab259d-kube-api-access-kf6px\") pod \"payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x\" (UID: \"b835588e-1042-46bb-8554-243c7dab259d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x"
Oct 31 01:06:43.469001 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.468917    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9mkf\" (UniqueName: \"kubernetes.io/projected/7549bb85-c79e-4740-b3f0-7f1c793b741a-kube-api-access-d9mkf\") pod \"payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl\" (UID: \"7549bb85-c79e-4740-b3f0-7f1c793b741a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl"
Oct 31 01:06:43.469176 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.469153    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7549bb85-c79e-4740-b3f0-7f1c793b741a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl\" (UID: \"7549bb85-c79e-4740-b3f0-7f1c793b741a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl"
Oct 31 01:06:43.500961 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.500931    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x"
Oct 31 01:06:43.572123 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.571465    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7549bb85-c79e-4740-b3f0-7f1c793b741a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl\" (UID: \"7549bb85-c79e-4740-b3f0-7f1c793b741a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl"
Oct 31 01:06:43.572123 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.571590    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9mkf\" (UniqueName: \"kubernetes.io/projected/7549bb85-c79e-4740-b3f0-7f1c793b741a-kube-api-access-d9mkf\") pod \"payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl\" (UID: \"7549bb85-c79e-4740-b3f0-7f1c793b741a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl"
Oct 31 01:06:43.572929 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.572897    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7549bb85-c79e-4740-b3f0-7f1c793b741a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl\" (UID: \"7549bb85-c79e-4740-b3f0-7f1c793b741a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl"
Oct 31 01:06:43.596401 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.596364    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9mkf\" (UniqueName: \"kubernetes.io/projected/7549bb85-c79e-4740-b3f0-7f1c793b741a-kube-api-access-d9mkf\") pod \"payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl\" (UID: \"7549bb85-c79e-4740-b3f0-7f1c793b741a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl"
Oct 31 01:06:43.725609 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.725492    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl"
Oct 31 01:06:43.823746 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.820317    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x"]
Oct 31 01:06:43.825346 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:06:43.824493    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb835588e_1042_46bb_8554_243c7dab259d.slice/crio-c2275dc6566b03da1469973109a2ce6b44529e9e1115d972e505c9d2fda64960 WatchSource:0}: Error finding container c2275dc6566b03da1469973109a2ce6b44529e9e1115d972e505c9d2fda64960: Status 404 returned error can't find the container with id c2275dc6566b03da1469973109a2ce6b44529e9e1115d972e505c9d2fda64960
Oct 31 01:06:43.948873 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:43.948842    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a206da9-cfc0-41d4-888c-1123aab28e5c" path="/var/lib/kubelet/pods/7a206da9-cfc0-41d4-888c-1123aab28e5c/volumes"
Oct 31 01:06:44.041942 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:06:44.041912    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7549bb85_c79e_4740_b3f0_7f1c793b741a.slice/crio-932e4819b18acf08d65834b52445671375661732c9426b152f906e334e26f1fd WatchSource:0}: Error finding container 932e4819b18acf08d65834b52445671375661732c9426b152f906e334e26f1fd: Status 404 returned error can't find the container with id 932e4819b18acf08d65834b52445671375661732c9426b152f906e334e26f1fd
Oct 31 01:06:44.042205 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:44.042182    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl"]
Oct 31 01:06:44.266920 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:44.266888    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl" event={"ID":"7549bb85-c79e-4740-b3f0-7f1c793b741a","Type":"ContainerStarted","Data":"5347aaec0f96079409844a760560609fe8056c79804deb47d9c00869cc00df49"}
Oct 31 01:06:44.267120 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:44.266941    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl" event={"ID":"7549bb85-c79e-4740-b3f0-7f1c793b741a","Type":"ContainerStarted","Data":"932e4819b18acf08d65834b52445671375661732c9426b152f906e334e26f1fd"}
Oct 31 01:06:44.267184 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:44.267140    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl"
Oct 31 01:06:44.268477 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:44.268450    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x" event={"ID":"b835588e-1042-46bb-8554-243c7dab259d","Type":"ContainerStarted","Data":"4140e3d7f9077107def2088220d601f2300748e23e1d547223716fd99eabc10d"}
Oct 31 01:06:44.268616 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:44.268489    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x" event={"ID":"b835588e-1042-46bb-8554-243c7dab259d","Type":"ContainerStarted","Data":"c2275dc6566b03da1469973109a2ce6b44529e9e1115d972e505c9d2fda64960"}
Oct 31 01:06:44.268695 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:44.268642    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x"
Oct 31 01:06:44.310546 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:44.310443    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl" podStartSLOduration=1.310429965 podStartE2EDuration="1.310429965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:44.307954303 +0000 UTC m=+2864.529989845" watchObservedRunningTime="2026-10-31 01:06:44.310429965 +0000 UTC m=+2864.532465506"
Oct 31 01:06:44.351636 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:44.351583    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x" podStartSLOduration=1.351567584 podStartE2EDuration="1.351567584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:44.349037347 +0000 UTC m=+2864.571072889" watchObservedRunningTime="2026-10-31 01:06:44.351567584 +0000 UTC m=+2864.573603139"
Oct 31 01:06:45.301741 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:06:45.301698    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:06:50.796306 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:50.796268    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"
Oct 31 01:06:50.809465 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:50.809439    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"
Oct 31 01:06:51.335068 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:51.335038    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"
Oct 31 01:06:55.179525 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:55.179493    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2"]
Oct 31 01:06:55.212226 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:55.212190    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2"]
Oct 31 01:06:55.280985 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:55.280956    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl"
Oct 31 01:06:55.281567 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:06:55.281540    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x"
Oct 31 01:07:00.180459 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:00.180408    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2" podUID="ed89b1a6-e362-43a8-bca0-1566333e8e9b" containerName="payload-processing" containerID="cri-o://c059dbbacb3b7f6aa23de561a58da5e153d0cb356e85955f157ada9f5464b1e4" gracePeriod=25
Oct 31 01:07:00.213203 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:00.213135    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2" podUID="e47fba4e-8b03-401e-9967-c6bdf6c78ec5" containerName="payload-pre-processing" containerID="cri-o://24ca92ad004132022e6293a67c2e0e39a4fce67ff59b0c324d3c14b95163186f" gracePeriod=25
Oct 31 01:07:07.919221 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:07.919180    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2" podUID="ed89b1a6-e362-43a8-bca0-1566333e8e9b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:07:07.919221 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:07.919201    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2" podUID="e47fba4e-8b03-401e-9967-c6bdf6c78ec5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:07:15.295997 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:07:15.295959    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:07:15.889876 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:15.889831    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"]
Oct 31 01:07:15.895621 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:15.895590    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:15.924765 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:15.913305    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-b8fd5a-openshift-default-dockercfg-mwm8c\""
Oct 31 01:07:15.931285 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:15.931257    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"]
Oct 31 01:07:16.027018 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.026967    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c28e892-156c-4df2-a55e-af21750b4fda-istiod-ca-cert\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.027204 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.027021    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-workload-socket\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.027204 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.027172    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-credential-socket\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.027204 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.027201    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-istio-envoy\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.027356 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.027230    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c28e892-156c-4df2-a55e-af21750b4fda-istio-podinfo\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.027356 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.027316    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c28e892-156c-4df2-a55e-af21750b4fda-istio-token\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.027356 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.027344    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-istio-data\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.027519 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.027490    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg9gq\" (UniqueName: \"kubernetes.io/projected/4c28e892-156c-4df2-a55e-af21750b4fda-kube-api-access-fg9gq\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.027630 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.027612    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-workload-certs\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.128957 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.128922    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-workload-certs\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.129131 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.129027    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c28e892-156c-4df2-a55e-af21750b4fda-istiod-ca-cert\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.129131 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.129048    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-workload-socket\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.129131 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.129088    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-credential-socket\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.129131 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.129110    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-istio-envoy\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.129330 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.129144    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c28e892-156c-4df2-a55e-af21750b4fda-istio-podinfo\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.129330 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.129198    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c28e892-156c-4df2-a55e-af21750b4fda-istio-token\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.129330 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.129227    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-istio-data\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.129497 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.129337    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg9gq\" (UniqueName: \"kubernetes.io/projected/4c28e892-156c-4df2-a55e-af21750b4fda-kube-api-access-fg9gq\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.129497 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.129365    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-workload-certs\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.129613 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.129520    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-workload-socket\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.129613 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.129567    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-credential-socket\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.129787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.129759    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-istio-data\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.129845 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.129817    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c28e892-156c-4df2-a55e-af21750b4fda-istiod-ca-cert\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.131614 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.131585    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-istio-envoy\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.131803 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.131785    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c28e892-156c-4df2-a55e-af21750b4fda-istio-podinfo\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.154159 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.154091    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c28e892-156c-4df2-a55e-af21750b4fda-istio-token\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.154289 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.154241    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg9gq\" (UniqueName: \"kubernetes.io/projected/4c28e892-156c-4df2-a55e-af21750b4fda-kube-api-access-fg9gq\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.235203 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.235171    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:16.555248 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:07:16.555211    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c28e892_156c_4df2_a55e_af21750b4fda.slice/crio-05a0ab5a1c336735c5200faa896c617cfe6035fe87191da1ddc765885deb993f WatchSource:0}: Error finding container 05a0ab5a1c336735c5200faa896c617cfe6035fe87191da1ddc765885deb993f: Status 404 returned error can't find the container with id 05a0ab5a1c336735c5200faa896c617cfe6035fe87191da1ddc765885deb993f
Oct 31 01:07:16.555964 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.555936    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"]
Oct 31 01:07:16.558028 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.557993    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:07:16.558156 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.558075    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:07:16.558156 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.558122    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:07:16.569084 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:16.569055    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7" event={"ID":"4c28e892-156c-4df2-a55e-af21750b4fda","Type":"ContainerStarted","Data":"05a0ab5a1c336735c5200faa896c617cfe6035fe87191da1ddc765885deb993f"}
Oct 31 01:07:17.577928 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:17.577885    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7" event={"ID":"4c28e892-156c-4df2-a55e-af21750b4fda","Type":"ContainerStarted","Data":"8b1e61b00df317da699357dcc29e787ad6a04e8268b1ead678c1a55ffc3e7f12"}
Oct 31 01:07:17.649027 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:17.648965    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7" podStartSLOduration=2.648947907 podStartE2EDuration="2.648947907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:17.645765811 +0000 UTC m=+2897.867801356" watchObservedRunningTime="2026-10-31 01:07:17.648947907 +0000 UTC m=+2897.870983449"
Oct 31 01:07:17.944166 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:17.944127    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2" podUID="e47fba4e-8b03-401e-9967-c6bdf6c78ec5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:07:17.944869 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:17.944843    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2" podUID="ed89b1a6-e362-43a8-bca0-1566333e8e9b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:07:18.240230 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:18.240080    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:18.243221 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:18.243184    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 31 01:07:18.243370 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:18.243250    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7" podUID="4c28e892-156c-4df2-a55e-af21750b4fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 31 01:07:19.237704 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:19.237144    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 31 01:07:19.237704 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:19.237210    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7" podUID="4c28e892-156c-4df2-a55e-af21750b4fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 31 01:07:20.242723 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:20.237252    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 31 01:07:20.242723 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:20.237338    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7" podUID="4c28e892-156c-4df2-a55e-af21750b4fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 31 01:07:20.595726 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:20.595452    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2"
Oct 31 01:07:20.605881 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:20.605848    4327 generic.go:356] "Generic (PLEG): container finished" podID="e47fba4e-8b03-401e-9967-c6bdf6c78ec5" containerID="24ca92ad004132022e6293a67c2e0e39a4fce67ff59b0c324d3c14b95163186f" exitCode=0
Oct 31 01:07:20.606055 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:20.605918    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2" event={"ID":"e47fba4e-8b03-401e-9967-c6bdf6c78ec5","Type":"ContainerDied","Data":"24ca92ad004132022e6293a67c2e0e39a4fce67ff59b0c324d3c14b95163186f"}
Oct 31 01:07:20.607532 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:20.607508    4327 generic.go:356] "Generic (PLEG): container finished" podID="ed89b1a6-e362-43a8-bca0-1566333e8e9b" containerID="c059dbbacb3b7f6aa23de561a58da5e153d0cb356e85955f157ada9f5464b1e4" exitCode=0
Oct 31 01:07:20.607707 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:20.607552    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2"
Oct 31 01:07:20.607707 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:20.607589    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2" event={"ID":"ed89b1a6-e362-43a8-bca0-1566333e8e9b","Type":"ContainerDied","Data":"c059dbbacb3b7f6aa23de561a58da5e153d0cb356e85955f157ada9f5464b1e4"}
Oct 31 01:07:20.607707 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:20.607646    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2" event={"ID":"ed89b1a6-e362-43a8-bca0-1566333e8e9b","Type":"ContainerDied","Data":"ce93c6913f3cd06b75663163332c13e19d0180e0e869191f85223b0b8c6c9562"}
Oct 31 01:07:20.607707 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:20.607691    4327 scope.go:122] "RemoveContainer" containerID="c059dbbacb3b7f6aa23de561a58da5e153d0cb356e85955f157ada9f5464b1e4"
Oct 31 01:07:20.648708 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:20.637040    4327 scope.go:122] "RemoveContainer" containerID="c059dbbacb3b7f6aa23de561a58da5e153d0cb356e85955f157ada9f5464b1e4"
Oct 31 01:07:20.653713 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:07:20.653401    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c059dbbacb3b7f6aa23de561a58da5e153d0cb356e85955f157ada9f5464b1e4\": container with ID starting with c059dbbacb3b7f6aa23de561a58da5e153d0cb356e85955f157ada9f5464b1e4 not found: ID does not exist" containerID="c059dbbacb3b7f6aa23de561a58da5e153d0cb356e85955f157ada9f5464b1e4"
Oct 31 01:07:20.653713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:20.653442    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c059dbbacb3b7f6aa23de561a58da5e153d0cb356e85955f157ada9f5464b1e4"} err="failed to get container status \"c059dbbacb3b7f6aa23de561a58da5e153d0cb356e85955f157ada9f5464b1e4\": rpc error: code = NotFound desc = could not find container \"c059dbbacb3b7f6aa23de561a58da5e153d0cb356e85955f157ada9f5464b1e4\": container with ID starting with c059dbbacb3b7f6aa23de561a58da5e153d0cb356e85955f157ada9f5464b1e4 not found: ID does not exist"
Oct 31 01:07:20.682648 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:20.682616    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2"
Oct 31 01:07:20.699016 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:20.698968    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed89b1a6-e362-43a8-bca0-1566333e8e9b-kube-api-access-wkhkf\" (UniqueName: \"kubernetes.io/projected/ed89b1a6-e362-43a8-bca0-1566333e8e9b-kube-api-access-wkhkf\") pod \"ed89b1a6-e362-43a8-bca0-1566333e8e9b\" (UID: \"ed89b1a6-e362-43a8-bca0-1566333e8e9b\") "
Oct 31 01:07:20.699193 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:20.699116    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed89b1a6-e362-43a8-bca0-1566333e8e9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed89b1a6-e362-43a8-bca0-1566333e8e9b-plugins-config-volume\") pod \"ed89b1a6-e362-43a8-bca0-1566333e8e9b\" (UID: \"ed89b1a6-e362-43a8-bca0-1566333e8e9b\") "
Oct 31 01:07:20.699998 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:20.699969    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed89b1a6-e362-43a8-bca0-1566333e8e9b-plugins-config-volume" pod "ed89b1a6-e362-43a8-bca0-1566333e8e9b" (UID: "ed89b1a6-e362-43a8-bca0-1566333e8e9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:07:20.702112 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:20.702091    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed89b1a6-e362-43a8-bca0-1566333e8e9b-kube-api-access-wkhkf" pod "ed89b1a6-e362-43a8-bca0-1566333e8e9b" (UID: "ed89b1a6-e362-43a8-bca0-1566333e8e9b"). InnerVolumeSpecName "kube-api-access-wkhkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:07:20.810291 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:20.800493    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e47fba4e-8b03-401e-9967-c6bdf6c78ec5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e47fba4e-8b03-401e-9967-c6bdf6c78ec5-plugins-config-volume\") pod \"e47fba4e-8b03-401e-9967-c6bdf6c78ec5\" (UID: \"e47fba4e-8b03-401e-9967-c6bdf6c78ec5\") "
Oct 31 01:07:20.810291 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:20.800932    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e47fba4e-8b03-401e-9967-c6bdf6c78ec5-kube-api-access-vw9j2\" (UniqueName: \"kubernetes.io/projected/e47fba4e-8b03-401e-9967-c6bdf6c78ec5-kube-api-access-vw9j2\") pod \"e47fba4e-8b03-401e-9967-c6bdf6c78ec5\" (UID: \"e47fba4e-8b03-401e-9967-c6bdf6c78ec5\") "
Oct 31 01:07:20.810291 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:20.804893    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkhkf\" (UniqueName: \"kubernetes.io/projected/ed89b1a6-e362-43a8-bca0-1566333e8e9b-kube-api-access-wkhkf\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:07:20.810291 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:20.804925    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed89b1a6-e362-43a8-bca0-1566333e8e9b-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:07:20.810291 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:20.806194    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e47fba4e-8b03-401e-9967-c6bdf6c78ec5-plugins-config-volume" pod "e47fba4e-8b03-401e-9967-c6bdf6c78ec5" (UID: "e47fba4e-8b03-401e-9967-c6bdf6c78ec5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:07:20.810734 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:20.810369    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e47fba4e-8b03-401e-9967-c6bdf6c78ec5-kube-api-access-vw9j2" pod "e47fba4e-8b03-401e-9967-c6bdf6c78ec5" (UID: "e47fba4e-8b03-401e-9967-c6bdf6c78ec5"). InnerVolumeSpecName "kube-api-access-vw9j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:07:20.907441 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:20.907404    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw9j2\" (UniqueName: \"kubernetes.io/projected/e47fba4e-8b03-401e-9967-c6bdf6c78ec5-kube-api-access-vw9j2\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:07:20.907441 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:20.907434    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e47fba4e-8b03-401e-9967-c6bdf6c78ec5-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:07:21.064306 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:21.064262    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2"]
Oct 31 01:07:21.074718 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:21.073789    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-57e594-86cdf6574c-dt5v2"]
Oct 31 01:07:21.239702 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:21.239309    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 31 01:07:21.239702 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:21.239369    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7" podUID="4c28e892-156c-4df2-a55e-af21750b4fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 31 01:07:21.679522 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:21.679491    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2"
Oct 31 01:07:21.680132 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:21.679501    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2" event={"ID":"e47fba4e-8b03-401e-9967-c6bdf6c78ec5","Type":"ContainerDied","Data":"973dd86705bc6f8037413bb6ceb61301b19387b81eb2525c62c34ac5b40fa9e4"}
Oct 31 01:07:21.680207 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:21.680192    4327 scope.go:122] "RemoveContainer" containerID="24ca92ad004132022e6293a67c2e0e39a4fce67ff59b0c324d3c14b95163186f"
Oct 31 01:07:21.746287 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:21.746112    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2"]
Oct 31 01:07:21.750189 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:21.750160    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-57e594-6784f95c8c-cw4m2"]
Oct 31 01:07:21.890901 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:21.890871    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"]
Oct 31 01:07:21.891096 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:21.891051    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9" podUID="20f13ccf-97ff-4e16-ba0b-c050043ce8c8" containerName="istio-proxy" containerID="cri-o://493baec8d2b60234b949a70d452589debc8a0c778ab3b7babd25782bae384e89" gracePeriod=30
Oct 31 01:07:21.950216 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:21.950139    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e47fba4e-8b03-401e-9967-c6bdf6c78ec5" path="/var/lib/kubelet/pods/e47fba4e-8b03-401e-9967-c6bdf6c78ec5/volumes"
Oct 31 01:07:21.950605 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:21.950590    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed89b1a6-e362-43a8-bca0-1566333e8e9b" path="/var/lib/kubelet/pods/ed89b1a6-e362-43a8-bca0-1566333e8e9b/volumes"
Oct 31 01:07:22.241761 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:22.241097    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 31 01:07:22.241761 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:22.241162    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7" podUID="4c28e892-156c-4df2-a55e-af21750b4fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 31 01:07:23.236705 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:23.235944    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 31 01:07:23.236705 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:23.236048    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7" podUID="4c28e892-156c-4df2-a55e-af21750b4fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 31 01:07:24.240983 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:24.240953    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:24.241470 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:24.241449    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:24.242240 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:24.242220    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:07:27.180433 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.180363    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 01:07:27.336024 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.335989    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-workload-certs\") pod \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") "
Oct 31 01:07:27.336210 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.336037    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-workload-socket\") pod \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") "
Oct 31 01:07:27.336210 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.336087    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-envoy\") pod \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") "
Oct 31 01:07:27.336210 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.336152    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-data\") pod \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") "
Oct 31 01:07:27.336321 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.336290    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-workload-certs" pod "20f13ccf-97ff-4e16-ba0b-c050043ce8c8" (UID: "20f13ccf-97ff-4e16-ba0b-c050043ce8c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:07:27.336321 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.336310    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-podinfo\") pod \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") "
Oct 31 01:07:27.336396 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.336307    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-workload-socket" pod "20f13ccf-97ff-4e16-ba0b-c050043ce8c8" (UID: "20f13ccf-97ff-4e16-ba0b-c050043ce8c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:07:27.336432 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.336413    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-kube-api-access-wlvx8\" (UniqueName: \"kubernetes.io/projected/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-kube-api-access-wlvx8\") pod \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") "
Oct 31 01:07:27.336493 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.336446    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-credential-socket\") pod \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") "
Oct 31 01:07:27.336493 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.336413    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-data" pod "20f13ccf-97ff-4e16-ba0b-c050043ce8c8" (UID: "20f13ccf-97ff-4e16-ba0b-c050043ce8c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:07:27.336493 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.336475    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istiod-ca-cert\") pod \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") "
Oct 31 01:07:27.336660 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.336528    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-token\" (UniqueName: \"kubernetes.io/projected/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-token\") pod \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\" (UID: \"20f13ccf-97ff-4e16-ba0b-c050043ce8c8\") "
Oct 31 01:07:27.336751 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.336731    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-credential-socket" pod "20f13ccf-97ff-4e16-ba0b-c050043ce8c8" (UID: "20f13ccf-97ff-4e16-ba0b-c050043ce8c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:07:27.336949 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.336922    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istiod-ca-cert" pod "20f13ccf-97ff-4e16-ba0b-c050043ce8c8" (UID: "20f13ccf-97ff-4e16-ba0b-c050043ce8c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:07:27.337235 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.337218    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:07:27.337235 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.337236    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:07:27.337409 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.337246    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:07:27.337409 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.337254    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:07:27.337409 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.337264    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:07:27.338788 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.338759    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-kube-api-access-wlvx8" pod "20f13ccf-97ff-4e16-ba0b-c050043ce8c8" (UID: "20f13ccf-97ff-4e16-ba0b-c050043ce8c8"). InnerVolumeSpecName "kube-api-access-wlvx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:07:27.338923 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.338835    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-envoy" pod "20f13ccf-97ff-4e16-ba0b-c050043ce8c8" (UID: "20f13ccf-97ff-4e16-ba0b-c050043ce8c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:07:27.338978 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.338957    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-podinfo" pod "20f13ccf-97ff-4e16-ba0b-c050043ce8c8" (UID: "20f13ccf-97ff-4e16-ba0b-c050043ce8c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:07:27.339035 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.339011    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-token" pod "20f13ccf-97ff-4e16-ba0b-c050043ce8c8" (UID: "20f13ccf-97ff-4e16-ba0b-c050043ce8c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:07:27.438947 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.438860    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:07:27.438947 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.438893    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlvx8\" (UniqueName: \"kubernetes.io/projected/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-kube-api-access-wlvx8\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:07:27.438947 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.438905    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:07:27.438947 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.438913    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20f13ccf-97ff-4e16-ba0b-c050043ce8c8-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:07:27.755740 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.755621    4327 generic.go:356] "Generic (PLEG): container finished" podID="20f13ccf-97ff-4e16-ba0b-c050043ce8c8" containerID="493baec8d2b60234b949a70d452589debc8a0c778ab3b7babd25782bae384e89" exitCode=0
Oct 31 01:07:27.755740 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.755674    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"
Oct 31 01:07:27.755969 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.755707    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9" event={"ID":"20f13ccf-97ff-4e16-ba0b-c050043ce8c8","Type":"ContainerDied","Data":"493baec8d2b60234b949a70d452589debc8a0c778ab3b7babd25782bae384e89"}
Oct 31 01:07:27.755969 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.755828    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9" event={"ID":"20f13ccf-97ff-4e16-ba0b-c050043ce8c8","Type":"ContainerDied","Data":"43b7b0848c285a2bdc7d5cc553530799a66902a48aee11b4929d88a3c4a58cab"}
Oct 31 01:07:27.755969 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.755860    4327 scope.go:122] "RemoveContainer" containerID="493baec8d2b60234b949a70d452589debc8a0c778ab3b7babd25782bae384e89"
Oct 31 01:07:27.768837 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.768795    4327 scope.go:122] "RemoveContainer" containerID="2ace50a97b7ca77da7c480b71ce106487ced0112d72fcb42d69e640c98049cad"
Oct 31 01:07:27.779869 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.779846    4327 scope.go:122] "RemoveContainer" containerID="493baec8d2b60234b949a70d452589debc8a0c778ab3b7babd25782bae384e89"
Oct 31 01:07:27.780240 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:07:27.780213    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"493baec8d2b60234b949a70d452589debc8a0c778ab3b7babd25782bae384e89\": container with ID starting with 493baec8d2b60234b949a70d452589debc8a0c778ab3b7babd25782bae384e89 not found: ID does not exist" containerID="493baec8d2b60234b949a70d452589debc8a0c778ab3b7babd25782bae384e89"
Oct 31 01:07:27.780324 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.780250    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"493baec8d2b60234b949a70d452589debc8a0c778ab3b7babd25782bae384e89"} err="failed to get container status \"493baec8d2b60234b949a70d452589debc8a0c778ab3b7babd25782bae384e89\": rpc error: code = NotFound desc = could not find container \"493baec8d2b60234b949a70d452589debc8a0c778ab3b7babd25782bae384e89\": container with ID starting with 493baec8d2b60234b949a70d452589debc8a0c778ab3b7babd25782bae384e89 not found: ID does not exist"
Oct 31 01:07:27.780324 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.780270    4327 scope.go:122] "RemoveContainer" containerID="2ace50a97b7ca77da7c480b71ce106487ced0112d72fcb42d69e640c98049cad"
Oct 31 01:07:27.780551 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:07:27.780533    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ace50a97b7ca77da7c480b71ce106487ced0112d72fcb42d69e640c98049cad\": container with ID starting with 2ace50a97b7ca77da7c480b71ce106487ced0112d72fcb42d69e640c98049cad not found: ID does not exist" containerID="2ace50a97b7ca77da7c480b71ce106487ced0112d72fcb42d69e640c98049cad"
Oct 31 01:07:27.780591 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.780558    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ace50a97b7ca77da7c480b71ce106487ced0112d72fcb42d69e640c98049cad"} err="failed to get container status \"2ace50a97b7ca77da7c480b71ce106487ced0112d72fcb42d69e640c98049cad\": rpc error: code = NotFound desc = could not find container \"2ace50a97b7ca77da7c480b71ce106487ced0112d72fcb42d69e640c98049cad\": container with ID starting with 2ace50a97b7ca77da7c480b71ce106487ced0112d72fcb42d69e640c98049cad not found: ID does not exist"
Oct 31 01:07:27.822716 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.822656    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"]
Oct 31 01:07:27.834961 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.834918    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-57e594-openshift-default-55cf466c9c-c8tb9"]
Oct 31 01:07:27.924787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:27.924755    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20f13ccf-97ff-4e16-ba0b-c050043ce8c8" path="/var/lib/kubelet/pods/20f13ccf-97ff-4e16-ba0b-c050043ce8c8/volumes"
Oct 31 01:07:33.008432 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:33.008396    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2"]
Oct 31 01:07:38.009281 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:38.009242    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2" podUID="881a9204-bc36-4e4f-bb7d-89715ac273fb" containerName="payload-pre-processing" containerID="cri-o://b4f2b4f1110172858e52446b1d97c8ccade36f41c63cf1710f0a12c3e6401b59" gracePeriod=25
Oct 31 01:07:45.294252 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:07:45.294208    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:07:45.393378 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:45.393343    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2" podUID="881a9204-bc36-4e4f-bb7d-89715ac273fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:07:51.860990 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:51.860950    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"]
Oct 31 01:07:55.393218 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:55.393184    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2" podUID="881a9204-bc36-4e4f-bb7d-89715ac273fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:07:58.345876 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:58.345847    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2"
Oct 31 01:07:58.477831 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:58.477801    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/881a9204-bc36-4e4f-bb7d-89715ac273fb-kube-api-access-5k8rw\" (UniqueName: \"kubernetes.io/projected/881a9204-bc36-4e4f-bb7d-89715ac273fb-kube-api-access-5k8rw\") pod \"881a9204-bc36-4e4f-bb7d-89715ac273fb\" (UID: \"881a9204-bc36-4e4f-bb7d-89715ac273fb\") "
Oct 31 01:07:58.478005 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:58.477995    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/881a9204-bc36-4e4f-bb7d-89715ac273fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/881a9204-bc36-4e4f-bb7d-89715ac273fb-plugins-config-volume\") pod \"881a9204-bc36-4e4f-bb7d-89715ac273fb\" (UID: \"881a9204-bc36-4e4f-bb7d-89715ac273fb\") "
Oct 31 01:07:58.478361 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:58.478333    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/881a9204-bc36-4e4f-bb7d-89715ac273fb-plugins-config-volume" pod "881a9204-bc36-4e4f-bb7d-89715ac273fb" (UID: "881a9204-bc36-4e4f-bb7d-89715ac273fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:07:58.479886 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:58.479867    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/881a9204-bc36-4e4f-bb7d-89715ac273fb-kube-api-access-5k8rw" pod "881a9204-bc36-4e4f-bb7d-89715ac273fb" (UID: "881a9204-bc36-4e4f-bb7d-89715ac273fb"). InnerVolumeSpecName "kube-api-access-5k8rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:07:58.594422 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:58.594335    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/881a9204-bc36-4e4f-bb7d-89715ac273fb-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:07:58.594658 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:58.594620    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k8rw\" (UniqueName: \"kubernetes.io/projected/881a9204-bc36-4e4f-bb7d-89715ac273fb-kube-api-access-5k8rw\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:07:58.983595 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:58.983557    4327 generic.go:356] "Generic (PLEG): container finished" podID="881a9204-bc36-4e4f-bb7d-89715ac273fb" containerID="b4f2b4f1110172858e52446b1d97c8ccade36f41c63cf1710f0a12c3e6401b59" exitCode=0
Oct 31 01:07:58.983828 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:58.983605    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2" event={"ID":"881a9204-bc36-4e4f-bb7d-89715ac273fb","Type":"ContainerDied","Data":"b4f2b4f1110172858e52446b1d97c8ccade36f41c63cf1710f0a12c3e6401b59"}
Oct 31 01:07:58.983828 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:58.983657    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2" event={"ID":"881a9204-bc36-4e4f-bb7d-89715ac273fb","Type":"ContainerDied","Data":"7fd62b1eef9fe7b40d1d9df6f38875ed138493d1c7f913ed307419e57e98ccbe"}
Oct 31 01:07:58.983828 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:58.983722    4327 scope.go:122] "RemoveContainer" containerID="b4f2b4f1110172858e52446b1d97c8ccade36f41c63cf1710f0a12c3e6401b59"
Oct 31 01:07:58.984011 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:58.983888    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2"
Oct 31 01:07:59.107087 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:59.103968    4327 scope.go:122] "RemoveContainer" containerID="b4f2b4f1110172858e52446b1d97c8ccade36f41c63cf1710f0a12c3e6401b59"
Oct 31 01:07:59.111607 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:07:59.110171    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4f2b4f1110172858e52446b1d97c8ccade36f41c63cf1710f0a12c3e6401b59\": container with ID starting with b4f2b4f1110172858e52446b1d97c8ccade36f41c63cf1710f0a12c3e6401b59 not found: ID does not exist" containerID="b4f2b4f1110172858e52446b1d97c8ccade36f41c63cf1710f0a12c3e6401b59"
Oct 31 01:07:59.111607 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:59.110223    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4f2b4f1110172858e52446b1d97c8ccade36f41c63cf1710f0a12c3e6401b59"} err="failed to get container status \"b4f2b4f1110172858e52446b1d97c8ccade36f41c63cf1710f0a12c3e6401b59\": rpc error: code = NotFound desc = could not find container \"b4f2b4f1110172858e52446b1d97c8ccade36f41c63cf1710f0a12c3e6401b59\": container with ID starting with b4f2b4f1110172858e52446b1d97c8ccade36f41c63cf1710f0a12c3e6401b59 not found: ID does not exist"
Oct 31 01:07:59.114782 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:59.113297    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2"]
Oct 31 01:07:59.122495 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:59.122468    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4f4e69-579b866c9-5rhl2"]
Oct 31 01:07:59.924518 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:07:59.924485    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="881a9204-bc36-4e4f-bb7d-89715ac273fb" path="/var/lib/kubelet/pods/881a9204-bc36-4e4f-bb7d-89715ac273fb/volumes"
Oct 31 01:08:04.987998 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:04.987964    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc"]
Oct 31 01:08:04.988674 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:04.988660    4327 cpu_manager.go:414] "containerMap: removing container" podUID="e47fba4e-8b03-401e-9967-c6bdf6c78ec5" containerName="payload-pre-processing"
Oct 31 01:08:04.988739 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:04.988690    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e47fba4e-8b03-401e-9967-c6bdf6c78ec5" containerName="payload-pre-processing"
Oct 31 01:08:04.988739 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:04.988713    4327 cpu_manager.go:414] "containerMap: removing container" podUID="20f13ccf-97ff-4e16-ba0b-c050043ce8c8" containerName="istio-proxy"
Oct 31 01:08:04.988739 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:04.988719    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20f13ccf-97ff-4e16-ba0b-c050043ce8c8" containerName="istio-proxy"
Oct 31 01:08:04.988739 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:04.988733    4327 cpu_manager.go:414] "containerMap: removing container" podUID="20f13ccf-97ff-4e16-ba0b-c050043ce8c8" containerName="istio-proxy"
Oct 31 01:08:04.988739 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:04.988739    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20f13ccf-97ff-4e16-ba0b-c050043ce8c8" containerName="istio-proxy"
Oct 31 01:08:04.988899 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:04.988748    4327 cpu_manager.go:414] "containerMap: removing container" podUID="881a9204-bc36-4e4f-bb7d-89715ac273fb" containerName="payload-pre-processing"
Oct 31 01:08:04.988899 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:04.988753    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="881a9204-bc36-4e4f-bb7d-89715ac273fb" containerName="payload-pre-processing"
Oct 31 01:08:04.988899 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:04.988765    4327 cpu_manager.go:414] "containerMap: removing container" podUID="ed89b1a6-e362-43a8-bca0-1566333e8e9b" containerName="payload-processing"
Oct 31 01:08:04.988899 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:04.988770    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed89b1a6-e362-43a8-bca0-1566333e8e9b" containerName="payload-processing"
Oct 31 01:08:04.988899 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:04.988866    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="20f13ccf-97ff-4e16-ba0b-c050043ce8c8" containerName="istio-proxy"
Oct 31 01:08:04.988899 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:04.988875    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="20f13ccf-97ff-4e16-ba0b-c050043ce8c8" containerName="istio-proxy"
Oct 31 01:08:04.988899 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:04.988884    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="881a9204-bc36-4e4f-bb7d-89715ac273fb" containerName="payload-pre-processing"
Oct 31 01:08:04.988899 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:04.988893    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed89b1a6-e362-43a8-bca0-1566333e8e9b" containerName="payload-processing"
Oct 31 01:08:04.989419 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:04.988903    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="e47fba4e-8b03-401e-9967-c6bdf6c78ec5" containerName="payload-pre-processing"
Oct 31 01:08:04.994046 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:04.994028    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc"
Oct 31 01:08:05.011731 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:05.011703    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc"]
Oct 31 01:08:05.019284 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:05.019257    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-ada0c1\""
Oct 31 01:08:05.074228 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:05.074194    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb"]
Oct 31 01:08:05.078880 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:05.078854    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb"
Oct 31 01:08:05.099869 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:05.099841    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-ada0c1-dockercfg-snrbr\""
Oct 31 01:08:05.107550 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:05.107512    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb"]
Oct 31 01:08:05.135989 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:05.134793    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz8nl\" (UniqueName: \"kubernetes.io/projected/0d05d663-30be-4ee3-97c8-801ab8d0f885-kube-api-access-rz8nl\") pod \"payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc\" (UID: \"0d05d663-30be-4ee3-97c8-801ab8d0f885\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc"
Oct 31 01:08:05.136603 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:05.136556    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d05d663-30be-4ee3-97c8-801ab8d0f885-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc\" (UID: \"0d05d663-30be-4ee3-97c8-801ab8d0f885\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc"
Oct 31 01:08:05.237808 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:05.237763    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d05d663-30be-4ee3-97c8-801ab8d0f885-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc\" (UID: \"0d05d663-30be-4ee3-97c8-801ab8d0f885\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc"
Oct 31 01:08:05.238083 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:05.237891    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rb5t\" (UniqueName: \"kubernetes.io/projected/2c725e03-2ab7-49a9-8251-606dc8f70d4f-kube-api-access-7rb5t\") pod \"payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb\" (UID: \"2c725e03-2ab7-49a9-8251-606dc8f70d4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb"
Oct 31 01:08:05.238083 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:05.237940    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz8nl\" (UniqueName: \"kubernetes.io/projected/0d05d663-30be-4ee3-97c8-801ab8d0f885-kube-api-access-rz8nl\") pod \"payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc\" (UID: \"0d05d663-30be-4ee3-97c8-801ab8d0f885\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc"
Oct 31 01:08:05.238270 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:05.238243    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c725e03-2ab7-49a9-8251-606dc8f70d4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb\" (UID: \"2c725e03-2ab7-49a9-8251-606dc8f70d4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb"
Oct 31 01:08:05.239362 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:05.239339    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d05d663-30be-4ee3-97c8-801ab8d0f885-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc\" (UID: \"0d05d663-30be-4ee3-97c8-801ab8d0f885\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc"
Oct 31 01:08:05.262042 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:05.262009    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz8nl\" (UniqueName: \"kubernetes.io/projected/0d05d663-30be-4ee3-97c8-801ab8d0f885-kube-api-access-rz8nl\") pod \"payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc\" (UID: \"0d05d663-30be-4ee3-97c8-801ab8d0f885\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc"
Oct 31 01:08:05.309326 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:05.309282    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc"
Oct 31 01:08:05.339884 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:05.339851    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rb5t\" (UniqueName: \"kubernetes.io/projected/2c725e03-2ab7-49a9-8251-606dc8f70d4f-kube-api-access-7rb5t\") pod \"payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb\" (UID: \"2c725e03-2ab7-49a9-8251-606dc8f70d4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb"
Oct 31 01:08:05.340063 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:05.340044    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c725e03-2ab7-49a9-8251-606dc8f70d4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb\" (UID: \"2c725e03-2ab7-49a9-8251-606dc8f70d4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb"
Oct 31 01:08:05.340875 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:05.340834    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c725e03-2ab7-49a9-8251-606dc8f70d4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb\" (UID: \"2c725e03-2ab7-49a9-8251-606dc8f70d4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb"
Oct 31 01:08:05.363666 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:05.363635    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rb5t\" (UniqueName: \"kubernetes.io/projected/2c725e03-2ab7-49a9-8251-606dc8f70d4f-kube-api-access-7rb5t\") pod \"payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb\" (UID: \"2c725e03-2ab7-49a9-8251-606dc8f70d4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb"
Oct 31 01:08:05.396728 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:05.395770    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb"
Oct 31 01:08:05.618004 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:08:05.617972    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d05d663_30be_4ee3_97c8_801ab8d0f885.slice/crio-b118eb8e194cf1ff04e04cf64b34f07a3751df09047346755cb26b039d015fec WatchSource:0}: Error finding container b118eb8e194cf1ff04e04cf64b34f07a3751df09047346755cb26b039d015fec: Status 404 returned error can't find the container with id b118eb8e194cf1ff04e04cf64b34f07a3751df09047346755cb26b039d015fec
Oct 31 01:08:05.619124 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:05.619103    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc"]
Oct 31 01:08:05.762070 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:05.750150    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb"]
Oct 31 01:08:06.043642 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:06.043542    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc" event={"ID":"0d05d663-30be-4ee3-97c8-801ab8d0f885","Type":"ContainerStarted","Data":"2e284c3bc5b4cfb9abf67cd2588e747e26e5d0406d1c088f8dc926484dc13127"}
Oct 31 01:08:06.043642 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:06.043608    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc" event={"ID":"0d05d663-30be-4ee3-97c8-801ab8d0f885","Type":"ContainerStarted","Data":"b118eb8e194cf1ff04e04cf64b34f07a3751df09047346755cb26b039d015fec"}
Oct 31 01:08:06.043642 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:06.043644    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc"
Oct 31 01:08:06.045577 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:06.045546    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb" event={"ID":"2c725e03-2ab7-49a9-8251-606dc8f70d4f","Type":"ContainerStarted","Data":"451ebdddadb823ba6ff721b3971825b3fa8263929be2fd584f61216bb89a239d"}
Oct 31 01:08:06.045768 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:06.045603    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb"
Oct 31 01:08:06.045768 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:06.045630    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb" event={"ID":"2c725e03-2ab7-49a9-8251-606dc8f70d4f","Type":"ContainerStarted","Data":"0aa4ccf6d8e49633568b98e56749ee067f2719c930c37b0005ea70ad3d96b073"}
Oct 31 01:08:06.087271 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:06.087217    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc" podStartSLOduration=2.087202974 podStartE2EDuration="2.087202974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:06.083770661 +0000 UTC m=+2946.305806205" watchObservedRunningTime="2026-10-31 01:08:06.087202974 +0000 UTC m=+2946.309238529"
Oct 31 01:08:06.127717 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:06.127585    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb" podStartSLOduration=1.127568066 podStartE2EDuration="1.127568066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:06.127061588 +0000 UTC m=+2946.349097127" watchObservedRunningTime="2026-10-31 01:08:06.127568066 +0000 UTC m=+2946.349603611"
Oct 31 01:08:06.885704 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:06.885621    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc" podUID="9abe1d77-158a-4806-85f5-954fc5854b06" containerName="main" containerID="cri-o://dfd897c8e48cefc309faf4fbe68d487be9f39f64026cf60ae7296fb3c6d3f141" gracePeriod=45
Oct 31 01:08:07.055987 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:07.055951    4327 generic.go:356] "Generic (PLEG): container finished" podID="9abe1d77-158a-4806-85f5-954fc5854b06" containerID="dfd897c8e48cefc309faf4fbe68d487be9f39f64026cf60ae7296fb3c6d3f141" exitCode=0
Oct 31 01:08:07.056401 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:07.056025    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc" event={"ID":"9abe1d77-158a-4806-85f5-954fc5854b06","Type":"ContainerDied","Data":"dfd897c8e48cefc309faf4fbe68d487be9f39f64026cf60ae7296fb3c6d3f141"}
Oct 31 01:08:07.267564 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:07.267535    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"
Oct 31 01:08:07.397223 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:07.397183    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-model-cache\") pod \"9abe1d77-158a-4806-85f5-954fc5854b06\" (UID: \"9abe1d77-158a-4806-85f5-954fc5854b06\") "
Oct 31 01:08:07.397428 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:07.397271    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-tmp-dir\") pod \"9abe1d77-158a-4806-85f5-954fc5854b06\" (UID: \"9abe1d77-158a-4806-85f5-954fc5854b06\") "
Oct 31 01:08:07.397428 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:07.397364    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-home\" (UniqueName: \"kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-home\") pod \"9abe1d77-158a-4806-85f5-954fc5854b06\" (UID: \"9abe1d77-158a-4806-85f5-954fc5854b06\") "
Oct 31 01:08:07.397428 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:07.397420    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-dshm\") pod \"9abe1d77-158a-4806-85f5-954fc5854b06\" (UID: \"9abe1d77-158a-4806-85f5-954fc5854b06\") "
Oct 31 01:08:07.397612 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:07.397452    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9abe1d77-158a-4806-85f5-954fc5854b06-tls-certs\" (UniqueName: \"kubernetes.io/secret/9abe1d77-158a-4806-85f5-954fc5854b06-tls-certs\") pod \"9abe1d77-158a-4806-85f5-954fc5854b06\" (UID: \"9abe1d77-158a-4806-85f5-954fc5854b06\") "
Oct 31 01:08:07.397612 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:07.397504    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-model-cache" pod "9abe1d77-158a-4806-85f5-954fc5854b06" (UID: "9abe1d77-158a-4806-85f5-954fc5854b06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:07.397612 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:07.397532    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9abe1d77-158a-4806-85f5-954fc5854b06-kube-api-access-r2s4z\" (UniqueName: \"kubernetes.io/projected/9abe1d77-158a-4806-85f5-954fc5854b06-kube-api-access-r2s4z\") pod \"9abe1d77-158a-4806-85f5-954fc5854b06\" (UID: \"9abe1d77-158a-4806-85f5-954fc5854b06\") "
Oct 31 01:08:07.397612 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:07.397595    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-tmp-dir" pod "9abe1d77-158a-4806-85f5-954fc5854b06" (UID: "9abe1d77-158a-4806-85f5-954fc5854b06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:07.398222 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:07.398200    4327 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-model-cache\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:08:07.398338 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:07.398225    4327 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-tmp-dir\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:08:07.398571 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:07.398551    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-home" pod "9abe1d77-158a-4806-85f5-954fc5854b06" (UID: "9abe1d77-158a-4806-85f5-954fc5854b06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:07.399854 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:07.399820    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-dshm" pod "9abe1d77-158a-4806-85f5-954fc5854b06" (UID: "9abe1d77-158a-4806-85f5-954fc5854b06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:07.400197 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:07.400164    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9abe1d77-158a-4806-85f5-954fc5854b06-tls-certs" pod "9abe1d77-158a-4806-85f5-954fc5854b06" (UID: "9abe1d77-158a-4806-85f5-954fc5854b06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:08:07.400285 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:07.400228    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abe1d77-158a-4806-85f5-954fc5854b06-kube-api-access-r2s4z" pod "9abe1d77-158a-4806-85f5-954fc5854b06" (UID: "9abe1d77-158a-4806-85f5-954fc5854b06"). InnerVolumeSpecName "kube-api-access-r2s4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:08:07.499519 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:07.499433    4327 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-home\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:08:07.499519 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:07.499463    4327 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9abe1d77-158a-4806-85f5-954fc5854b06-dshm\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:08:07.499519 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:07.499472    4327 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9abe1d77-158a-4806-85f5-954fc5854b06-tls-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:08:07.499519 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:07.499482    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2s4z\" (UniqueName: \"kubernetes.io/projected/9abe1d77-158a-4806-85f5-954fc5854b06-kube-api-access-r2s4z\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:08:08.065499 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:08.065458    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"
Oct 31 01:08:08.065952 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:08.065459    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc" event={"ID":"9abe1d77-158a-4806-85f5-954fc5854b06","Type":"ContainerDied","Data":"058624440cec8b6f586f1687d39e948f047de543081a45f25c274f0daf86c51c"}
Oct 31 01:08:08.065952 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:08.065604    4327 scope.go:122] "RemoveContainer" containerID="dfd897c8e48cefc309faf4fbe68d487be9f39f64026cf60ae7296fb3c6d3f141"
Oct 31 01:08:08.126194 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:08.126150    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"]
Oct 31 01:08:08.139764 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:08.139706    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-6f8856/test-model-6f8856-kserve-f767f9b45-7qmlc"]
Oct 31 01:08:09.922455 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:09.922423    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9abe1d77-158a-4806-85f5-954fc5854b06" path="/var/lib/kubelet/pods/9abe1d77-158a-4806-85f5-954fc5854b06/volumes"
Oct 31 01:08:15.293994 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:08:15.293950    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:08:17.059221 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:17.059192    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb"
Oct 31 01:08:17.059633 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:17.059266    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc"
Oct 31 01:08:27.841862 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:27.841003    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl"]
Oct 31 01:08:27.865022 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:27.864976    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt"]
Oct 31 01:08:28.114963 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.114927    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"]
Oct 31 01:08:28.115738 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.115721    4327 cpu_manager.go:414] "containerMap: removing container" podUID="9abe1d77-158a-4806-85f5-954fc5854b06" containerName="main"
Oct 31 01:08:28.115738 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.115738    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9abe1d77-158a-4806-85f5-954fc5854b06" containerName="main"
Oct 31 01:08:28.115880 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.115871    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="9abe1d77-158a-4806-85f5-954fc5854b06" containerName="main"
Oct 31 01:08:28.120497 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.120460    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.139613 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.139222    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6f8856-openshift-default-dockercfg-k84qx\""
Oct 31 01:08:28.143568 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.143541    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"]
Oct 31 01:08:28.216266 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.216223    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-workload-socket\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.216472 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.216424    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-podinfo\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.216549 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.216475    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-envoy\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.216697 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.216648    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-workload-certs\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.216856 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.216720    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-credential-socket\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.216856 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.216835    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istiod-ca-cert\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.217009 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.216875    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-data\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.217009 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.216956    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zghtg\" (UniqueName: \"kubernetes.io/projected/46567832-e7e5-42f1-be7b-0bfec97b6b0b-kube-api-access-zghtg\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.217129 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.217054    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-token\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.317868 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.317834    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-token\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.318068 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.317891    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-workload-socket\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.318068 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.317974    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-podinfo\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.318068 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.318004    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-envoy\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.318241 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.318135    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-workload-certs\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.318241 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.318175    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-credential-socket\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.318428 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.318406    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istiod-ca-cert\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.318499 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.318457    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-data\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.318499 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.318491    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zghtg\" (UniqueName: \"kubernetes.io/projected/46567832-e7e5-42f1-be7b-0bfec97b6b0b-kube-api-access-zghtg\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.318607 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.318520    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-workload-socket\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.319279 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.319064    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-data\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.319279 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.319121    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-workload-certs\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.319279 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.319143    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-credential-socket\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.319527 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.319379    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istiod-ca-cert\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.320833 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.320809    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-envoy\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.321480 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.321457    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-podinfo\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.341650 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.341605    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-token\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.341796 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.341668    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zghtg\" (UniqueName: \"kubernetes.io/projected/46567832-e7e5-42f1-be7b-0bfec97b6b0b-kube-api-access-zghtg\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.436290 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.436208    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:28.836154 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:08:28.836119    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46567832_e7e5_42f1_be7b_0bfec97b6b0b.slice/crio-88394510cd9b75b6ebb3ec952e7f77f5773c03d1f555dca374cb794ba0552835 WatchSource:0}: Error finding container 88394510cd9b75b6ebb3ec952e7f77f5773c03d1f555dca374cb794ba0552835: Status 404 returned error can't find the container with id 88394510cd9b75b6ebb3ec952e7f77f5773c03d1f555dca374cb794ba0552835
Oct 31 01:08:28.838054 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.838032    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"]
Oct 31 01:08:28.839107 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.839070    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:08:28.839202 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.839138    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:08:28.839202 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:28.839167    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:08:29.233256 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:29.233215    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4" event={"ID":"46567832-e7e5-42f1-be7b-0bfec97b6b0b","Type":"ContainerStarted","Data":"44ff5e3660ffd3fcc6879a8028ca89230bf11e97e4e2d2cc893d492b30e1ac89"}
Oct 31 01:08:29.233644 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:29.233282    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4" event={"ID":"46567832-e7e5-42f1-be7b-0bfec97b6b0b","Type":"ContainerStarted","Data":"88394510cd9b75b6ebb3ec952e7f77f5773c03d1f555dca374cb794ba0552835"}
Oct 31 01:08:29.281823 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:29.281763    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4" podStartSLOduration=1.281748331 podStartE2EDuration="1.281748331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:29.278885694 +0000 UTC m=+2969.500921237" watchObservedRunningTime="2026-10-31 01:08:29.281748331 +0000 UTC m=+2969.503783873"
Oct 31 01:08:29.437192 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:29.437159    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:29.439062 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:29.439029    4327 patch_prober.go:29] interesting pod/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 31 01:08:29.439229 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:29.439096    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4" podUID="46567832-e7e5-42f1-be7b-0bfec97b6b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 31 01:08:30.437702 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:30.437126    4327 patch_prober.go:29] interesting pod/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 31 01:08:30.437702 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:30.437192    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4" podUID="46567832-e7e5-42f1-be7b-0bfec97b6b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 31 01:08:31.438479 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:31.438439    4327 patch_prober.go:29] interesting pod/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 31 01:08:31.439130 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:31.439076    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4" podUID="46567832-e7e5-42f1-be7b-0bfec97b6b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 31 01:08:32.440119 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:32.440078    4327 patch_prober.go:29] interesting pod/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 31 01:08:32.440756 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:32.440722    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4" podUID="46567832-e7e5-42f1-be7b-0bfec97b6b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 31 01:08:32.875700 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:32.873247    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl" podUID="cc06dda6-d818-4a58-89d3-8eb40043b464" containerName="payload-processing" containerID="cri-o://1ef143008ef5a7d320334b1bb9ebb9298d92e04ae041bdbf83dc2a8acd96f339" gracePeriod=25
Oct 31 01:08:32.875929 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:32.875871    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt" podUID="4ee1484c-003a-4b17-af4a-830ca58a1691" containerName="payload-pre-processing" containerID="cri-o://4d3506607085a398122bbc0dac327bdb6d3a467b6888eaf359ffb904282be4ba" gracePeriod=25
Oct 31 01:08:33.198937 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:33.198603    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl"
Oct 31 01:08:33.290998 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:33.290957    4327 generic.go:356] "Generic (PLEG): container finished" podID="cc06dda6-d818-4a58-89d3-8eb40043b464" containerID="1ef143008ef5a7d320334b1bb9ebb9298d92e04ae041bdbf83dc2a8acd96f339" exitCode=0
Oct 31 01:08:33.291176 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:33.291049    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl" event={"ID":"cc06dda6-d818-4a58-89d3-8eb40043b464","Type":"ContainerDied","Data":"1ef143008ef5a7d320334b1bb9ebb9298d92e04ae041bdbf83dc2a8acd96f339"}
Oct 31 01:08:33.291176 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:33.291101    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl" event={"ID":"cc06dda6-d818-4a58-89d3-8eb40043b464","Type":"ContainerDied","Data":"ea95a7a1ae759f4c376668571e807cedfef8ce7a3fd48b0ef303e38b7f56dc4c"}
Oct 31 01:08:33.291176 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:33.291131    4327 scope.go:122] "RemoveContainer" containerID="1ef143008ef5a7d320334b1bb9ebb9298d92e04ae041bdbf83dc2a8acd96f339"
Oct 31 01:08:33.291311 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:33.291294    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl"
Oct 31 01:08:33.302210 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:33.302169    4327 scope.go:122] "RemoveContainer" containerID="1ef143008ef5a7d320334b1bb9ebb9298d92e04ae041bdbf83dc2a8acd96f339"
Oct 31 01:08:33.302438 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:08:33.302419    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ef143008ef5a7d320334b1bb9ebb9298d92e04ae041bdbf83dc2a8acd96f339\": container with ID starting with 1ef143008ef5a7d320334b1bb9ebb9298d92e04ae041bdbf83dc2a8acd96f339 not found: ID does not exist" containerID="1ef143008ef5a7d320334b1bb9ebb9298d92e04ae041bdbf83dc2a8acd96f339"
Oct 31 01:08:33.302511 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:33.302445    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef143008ef5a7d320334b1bb9ebb9298d92e04ae041bdbf83dc2a8acd96f339"} err="failed to get container status \"1ef143008ef5a7d320334b1bb9ebb9298d92e04ae041bdbf83dc2a8acd96f339\": rpc error: code = NotFound desc = could not find container \"1ef143008ef5a7d320334b1bb9ebb9298d92e04ae041bdbf83dc2a8acd96f339\": container with ID starting with 1ef143008ef5a7d320334b1bb9ebb9298d92e04ae041bdbf83dc2a8acd96f339 not found: ID does not exist"
Oct 31 01:08:33.306311 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:33.306291    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc06dda6-d818-4a58-89d3-8eb40043b464-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc06dda6-d818-4a58-89d3-8eb40043b464-plugins-config-volume\") pod \"cc06dda6-d818-4a58-89d3-8eb40043b464\" (UID: \"cc06dda6-d818-4a58-89d3-8eb40043b464\") "
Oct 31 01:08:33.306786 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:33.306464    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc06dda6-d818-4a58-89d3-8eb40043b464-kube-api-access-r4cf6\" (UniqueName: \"kubernetes.io/projected/cc06dda6-d818-4a58-89d3-8eb40043b464-kube-api-access-r4cf6\") pod \"cc06dda6-d818-4a58-89d3-8eb40043b464\" (UID: \"cc06dda6-d818-4a58-89d3-8eb40043b464\") "
Oct 31 01:08:33.307010 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:33.306984    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc06dda6-d818-4a58-89d3-8eb40043b464-plugins-config-volume" pod "cc06dda6-d818-4a58-89d3-8eb40043b464" (UID: "cc06dda6-d818-4a58-89d3-8eb40043b464"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:08:33.308889 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:33.308861    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc06dda6-d818-4a58-89d3-8eb40043b464-kube-api-access-r4cf6" pod "cc06dda6-d818-4a58-89d3-8eb40043b464" (UID: "cc06dda6-d818-4a58-89d3-8eb40043b464"). InnerVolumeSpecName "kube-api-access-r4cf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:08:33.409081 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:33.408993    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4cf6\" (UniqueName: \"kubernetes.io/projected/cc06dda6-d818-4a58-89d3-8eb40043b464-kube-api-access-r4cf6\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:08:33.409081 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:33.409031    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc06dda6-d818-4a58-89d3-8eb40043b464-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:08:33.441705 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:33.437064    4327 patch_prober.go:29] interesting pod/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 31 01:08:33.441705 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:33.437122    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4" podUID="46567832-e7e5-42f1-be7b-0bfec97b6b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 31 01:08:33.652377 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:33.652338    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl"]
Oct 31 01:08:33.657705 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:33.657658    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f8856-647b85df9c-tbspl"]
Oct 31 01:08:33.950347 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:33.950253    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc06dda6-d818-4a58-89d3-8eb40043b464" path="/var/lib/kubelet/pods/cc06dda6-d818-4a58-89d3-8eb40043b464/volumes"
Oct 31 01:08:34.439145 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:34.439108    4327 patch_prober.go:29] interesting pod/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 31 01:08:34.439466 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:34.439421    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4" podUID="46567832-e7e5-42f1-be7b-0bfec97b6b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 31 01:08:35.391303 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:35.391262    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt" podUID="4ee1484c-003a-4b17-af4a-830ca58a1691" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:08:35.437396 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:35.437350    4327 patch_prober.go:29] interesting pod/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 31 01:08:35.437577 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:35.437429    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4" podUID="46567832-e7e5-42f1-be7b-0bfec97b6b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 31 01:08:36.441874 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:36.441843    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:36.442387 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:36.442288    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:36.442962 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:36.442942    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:08:45.293539 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:08:45.293499    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:08:45.390701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:45.390637    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt" podUID="4ee1484c-003a-4b17-af4a-830ca58a1691" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:08:51.847441 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:51.847407    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"]
Oct 31 01:08:51.848243 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:51.848222    4327 cpu_manager.go:414] "containerMap: removing container" podUID="cc06dda6-d818-4a58-89d3-8eb40043b464" containerName="payload-processing"
Oct 31 01:08:51.848243 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:51.848242    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc06dda6-d818-4a58-89d3-8eb40043b464" containerName="payload-processing"
Oct 31 01:08:51.848407 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:51.848389    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc06dda6-d818-4a58-89d3-8eb40043b464" containerName="payload-processing"
Oct 31 01:08:51.853535 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:51.853512    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"
Oct 31 01:08:51.872394 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:51.872358    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"]
Oct 31 01:08:51.881170 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:51.880286    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-714f8f\"/\"kube-root-ca.crt\""
Oct 31 01:08:51.881170 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:51.880379    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-714f8f\"/\"openshift-service-ca.crt\""
Oct 31 01:08:51.881170 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:51.880487    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-714f8f\"/\"default-dockercfg-b6ssk\""
Oct 31 01:08:51.883069 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:51.883042    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-714f8f\"/\"rate-test-model-714f8f-kserve-self-signed-certs\""
Oct 31 01:08:52.021080 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:52.021046    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-home\") pod \"rate-test-model-714f8f-kserve-6c686b7549-5gmk8\" (UID: \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\") " pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"
Oct 31 01:08:52.021262 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:52.021107    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-682ws\" (UniqueName: \"kubernetes.io/projected/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-kube-api-access-682ws\") pod \"rate-test-model-714f8f-kserve-6c686b7549-5gmk8\" (UID: \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\") " pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"
Oct 31 01:08:52.021262 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:52.021222    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-tmp-dir\") pod \"rate-test-model-714f8f-kserve-6c686b7549-5gmk8\" (UID: \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\") " pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"
Oct 31 01:08:52.021427 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:52.021406    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-model-cache\") pod \"rate-test-model-714f8f-kserve-6c686b7549-5gmk8\" (UID: \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\") " pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"
Oct 31 01:08:52.021588 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:52.021546    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-dshm\") pod \"rate-test-model-714f8f-kserve-6c686b7549-5gmk8\" (UID: \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\") " pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"
Oct 31 01:08:52.021970 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:52.021940    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-tls-certs\") pod \"rate-test-model-714f8f-kserve-6c686b7549-5gmk8\" (UID: \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\") " pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"
Oct 31 01:08:52.123721 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:52.123650    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-tls-certs\") pod \"rate-test-model-714f8f-kserve-6c686b7549-5gmk8\" (UID: \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\") " pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"
Oct 31 01:08:52.124035 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:52.124013    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-home\") pod \"rate-test-model-714f8f-kserve-6c686b7549-5gmk8\" (UID: \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\") " pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"
Oct 31 01:08:52.124198 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:52.124182    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-682ws\" (UniqueName: \"kubernetes.io/projected/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-kube-api-access-682ws\") pod \"rate-test-model-714f8f-kserve-6c686b7549-5gmk8\" (UID: \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\") " pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"
Oct 31 01:08:52.124314 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:52.124302    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-tmp-dir\") pod \"rate-test-model-714f8f-kserve-6c686b7549-5gmk8\" (UID: \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\") " pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"
Oct 31 01:08:52.124481 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:52.124469    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-model-cache\") pod \"rate-test-model-714f8f-kserve-6c686b7549-5gmk8\" (UID: \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\") " pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"
Oct 31 01:08:52.124663 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:52.124651    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-dshm\") pod \"rate-test-model-714f8f-kserve-6c686b7549-5gmk8\" (UID: \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\") " pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"
Oct 31 01:08:52.126385 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:52.125820    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-tmp-dir\") pod \"rate-test-model-714f8f-kserve-6c686b7549-5gmk8\" (UID: \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\") " pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"
Oct 31 01:08:52.126385 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:52.126074    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-model-cache\") pod \"rate-test-model-714f8f-kserve-6c686b7549-5gmk8\" (UID: \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\") " pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"
Oct 31 01:08:52.127901 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:52.127320    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-home\") pod \"rate-test-model-714f8f-kserve-6c686b7549-5gmk8\" (UID: \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\") " pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"
Oct 31 01:08:52.148731 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:52.148383    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-tls-certs\") pod \"rate-test-model-714f8f-kserve-6c686b7549-5gmk8\" (UID: \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\") " pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"
Oct 31 01:08:52.150640 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:52.150603    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-dshm\") pod \"rate-test-model-714f8f-kserve-6c686b7549-5gmk8\" (UID: \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\") " pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"
Oct 31 01:08:52.151384 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:52.151359    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-682ws\" (UniqueName: \"kubernetes.io/projected/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-kube-api-access-682ws\") pod \"rate-test-model-714f8f-kserve-6c686b7549-5gmk8\" (UID: \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\") " pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"
Oct 31 01:08:52.169456 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:52.169411    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"
Oct 31 01:08:52.481154 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:08:52.481113    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d2ed373_a68e_4048_a3d4_e3bcee6f290d.slice/crio-54da1b58b0ff6511c0709b74e92c0e38522329deec162df8475822c046bbff88 WatchSource:0}: Error finding container 54da1b58b0ff6511c0709b74e92c0e38522329deec162df8475822c046bbff88: Status 404 returned error can't find the container with id 54da1b58b0ff6511c0709b74e92c0e38522329deec162df8475822c046bbff88
Oct 31 01:08:52.481749 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:52.481646    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"]
Oct 31 01:08:53.169524 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:53.169488    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt"
Oct 31 01:08:53.342906 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:53.342870    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ee1484c-003a-4b17-af4a-830ca58a1691-kube-api-access-5vvwn\" (UniqueName: \"kubernetes.io/projected/4ee1484c-003a-4b17-af4a-830ca58a1691-kube-api-access-5vvwn\") pod \"4ee1484c-003a-4b17-af4a-830ca58a1691\" (UID: \"4ee1484c-003a-4b17-af4a-830ca58a1691\") "
Oct 31 01:08:53.343095 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:53.343082    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ee1484c-003a-4b17-af4a-830ca58a1691-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ee1484c-003a-4b17-af4a-830ca58a1691-plugins-config-volume\") pod \"4ee1484c-003a-4b17-af4a-830ca58a1691\" (UID: \"4ee1484c-003a-4b17-af4a-830ca58a1691\") "
Oct 31 01:08:53.343453 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:53.343425    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ee1484c-003a-4b17-af4a-830ca58a1691-plugins-config-volume" pod "4ee1484c-003a-4b17-af4a-830ca58a1691" (UID: "4ee1484c-003a-4b17-af4a-830ca58a1691"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:08:53.343591 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:53.343574    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ee1484c-003a-4b17-af4a-830ca58a1691-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:08:53.345154 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:53.345129    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee1484c-003a-4b17-af4a-830ca58a1691-kube-api-access-5vvwn" pod "4ee1484c-003a-4b17-af4a-830ca58a1691" (UID: "4ee1484c-003a-4b17-af4a-830ca58a1691"). InnerVolumeSpecName "kube-api-access-5vvwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:08:53.445535 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:53.445450    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vvwn\" (UniqueName: \"kubernetes.io/projected/4ee1484c-003a-4b17-af4a-830ca58a1691-kube-api-access-5vvwn\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:08:53.454432 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:53.454396    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8" event={"ID":"1d2ed373-a68e-4048-a3d4-e3bcee6f290d","Type":"ContainerStarted","Data":"02cf95672e17830b978fee45e8adc3d643bca03157876057122d15315d0f5b05"}
Oct 31 01:08:53.454631 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:53.454444    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8" event={"ID":"1d2ed373-a68e-4048-a3d4-e3bcee6f290d","Type":"ContainerStarted","Data":"54da1b58b0ff6511c0709b74e92c0e38522329deec162df8475822c046bbff88"}
Oct 31 01:08:53.455904 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:53.455876    4327 generic.go:356] "Generic (PLEG): container finished" podID="4ee1484c-003a-4b17-af4a-830ca58a1691" containerID="4d3506607085a398122bbc0dac327bdb6d3a467b6888eaf359ffb904282be4ba" exitCode=0
Oct 31 01:08:53.456005 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:53.455915    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt"
Oct 31 01:08:53.456005 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:53.455940    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt" event={"ID":"4ee1484c-003a-4b17-af4a-830ca58a1691","Type":"ContainerDied","Data":"4d3506607085a398122bbc0dac327bdb6d3a467b6888eaf359ffb904282be4ba"}
Oct 31 01:08:53.456005 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:53.455984    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt" event={"ID":"4ee1484c-003a-4b17-af4a-830ca58a1691","Type":"ContainerDied","Data":"086ef5af30b9bb6c6aa3ee95d2224e8e7e733b6b5d1d2b825ad6f541403cb3ac"}
Oct 31 01:08:53.456122 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:53.456016    4327 scope.go:122] "RemoveContainer" containerID="4d3506607085a398122bbc0dac327bdb6d3a467b6888eaf359ffb904282be4ba"
Oct 31 01:08:53.467250 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:53.467224    4327 scope.go:122] "RemoveContainer" containerID="4d3506607085a398122bbc0dac327bdb6d3a467b6888eaf359ffb904282be4ba"
Oct 31 01:08:53.467610 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:08:53.467588    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d3506607085a398122bbc0dac327bdb6d3a467b6888eaf359ffb904282be4ba\": container with ID starting with 4d3506607085a398122bbc0dac327bdb6d3a467b6888eaf359ffb904282be4ba not found: ID does not exist" containerID="4d3506607085a398122bbc0dac327bdb6d3a467b6888eaf359ffb904282be4ba"
Oct 31 01:08:53.467700 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:53.467618    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d3506607085a398122bbc0dac327bdb6d3a467b6888eaf359ffb904282be4ba"} err="failed to get container status \"4d3506607085a398122bbc0dac327bdb6d3a467b6888eaf359ffb904282be4ba\": rpc error: code = NotFound desc = could not find container \"4d3506607085a398122bbc0dac327bdb6d3a467b6888eaf359ffb904282be4ba\": container with ID starting with 4d3506607085a398122bbc0dac327bdb6d3a467b6888eaf359ffb904282be4ba not found: ID does not exist"
Oct 31 01:08:53.504713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:53.504645    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8" podStartSLOduration=2.50463231 podStartE2EDuration="2.50463231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:53.503155452 +0000 UTC m=+2993.725190995" watchObservedRunningTime="2026-10-31 01:08:53.50463231 +0000 UTC m=+2993.726667851"
Oct 31 01:08:53.547185 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:53.546433    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt"]
Oct 31 01:08:53.555209 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:53.555177    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f8856-9c858b495-pgqmt"]
Oct 31 01:08:53.923147 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:53.923109    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ee1484c-003a-4b17-af4a-830ca58a1691" path="/var/lib/kubelet/pods/4ee1484c-003a-4b17-af4a-830ca58a1691/volumes"
Oct 31 01:08:54.169977 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:54.169945    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"
Oct 31 01:08:55.424145 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:55.424112    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z"]
Oct 31 01:08:55.424645 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:55.424299    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z" podUID="8c9dee08-2aa4-4c88-bea7-385a01d519bd" containerName="maas-api" containerID="cri-o://47f2d0934c937efe9e4f16e32e72aecdc84f1840aa5a5ee1f662a3cc80c95884" gracePeriod=30
Oct 31 01:08:55.789716 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:55.789688    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z"
Oct 31 01:08:55.835094 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:55.835056    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c9dee08-2aa4-4c88-bea7-385a01d519bd-kube-api-access-snrcn\" (UniqueName: \"kubernetes.io/projected/8c9dee08-2aa4-4c88-bea7-385a01d519bd-kube-api-access-snrcn\") pod \"8c9dee08-2aa4-4c88-bea7-385a01d519bd\" (UID: \"8c9dee08-2aa4-4c88-bea7-385a01d519bd\") "
Oct 31 01:08:55.835532 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:55.835494    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c9dee08-2aa4-4c88-bea7-385a01d519bd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c9dee08-2aa4-4c88-bea7-385a01d519bd-openshift-service-ca\") pod \"8c9dee08-2aa4-4c88-bea7-385a01d519bd\" (UID: \"8c9dee08-2aa4-4c88-bea7-385a01d519bd\") "
Oct 31 01:08:55.835739 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:55.835695    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c9dee08-2aa4-4c88-bea7-385a01d519bd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c9dee08-2aa4-4c88-bea7-385a01d519bd-maas-api-tls\") pod \"8c9dee08-2aa4-4c88-bea7-385a01d519bd\" (UID: \"8c9dee08-2aa4-4c88-bea7-385a01d519bd\") "
Oct 31 01:08:55.835842 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:55.835750    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c9dee08-2aa4-4c88-bea7-385a01d519bd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c9dee08-2aa4-4c88-bea7-385a01d519bd-metrics-certs\") pod \"8c9dee08-2aa4-4c88-bea7-385a01d519bd\" (UID: \"8c9dee08-2aa4-4c88-bea7-385a01d519bd\") "
Oct 31 01:08:55.839708 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:55.839377    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c9dee08-2aa4-4c88-bea7-385a01d519bd-openshift-service-ca" pod "8c9dee08-2aa4-4c88-bea7-385a01d519bd" (UID: "8c9dee08-2aa4-4c88-bea7-385a01d519bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:08:55.850881 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:55.850555    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c9dee08-2aa4-4c88-bea7-385a01d519bd-kube-api-access-snrcn" pod "8c9dee08-2aa4-4c88-bea7-385a01d519bd" (UID: "8c9dee08-2aa4-4c88-bea7-385a01d519bd"). InnerVolumeSpecName "kube-api-access-snrcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:08:55.859703 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:55.857129    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c9dee08-2aa4-4c88-bea7-385a01d519bd-metrics-certs" pod "8c9dee08-2aa4-4c88-bea7-385a01d519bd" (UID: "8c9dee08-2aa4-4c88-bea7-385a01d519bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:08:55.859703 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:55.858164    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c9dee08-2aa4-4c88-bea7-385a01d519bd-maas-api-tls" pod "8c9dee08-2aa4-4c88-bea7-385a01d519bd" (UID: "8c9dee08-2aa4-4c88-bea7-385a01d519bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:08:55.939301 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:55.939268    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c9dee08-2aa4-4c88-bea7-385a01d519bd-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:08:55.939301 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:55.939302    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c9dee08-2aa4-4c88-bea7-385a01d519bd-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:08:55.939552 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:55.939312    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c9dee08-2aa4-4c88-bea7-385a01d519bd-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:08:55.939552 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:55.939320    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snrcn\" (UniqueName: \"kubernetes.io/projected/8c9dee08-2aa4-4c88-bea7-385a01d519bd-kube-api-access-snrcn\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:08:56.481641 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:56.481607    4327 generic.go:356] "Generic (PLEG): container finished" podID="8c9dee08-2aa4-4c88-bea7-385a01d519bd" containerID="47f2d0934c937efe9e4f16e32e72aecdc84f1840aa5a5ee1f662a3cc80c95884" exitCode=0
Oct 31 01:08:56.482114 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:56.481651    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z"
Oct 31 01:08:56.482114 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:56.481692    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z" event={"ID":"8c9dee08-2aa4-4c88-bea7-385a01d519bd","Type":"ContainerDied","Data":"47f2d0934c937efe9e4f16e32e72aecdc84f1840aa5a5ee1f662a3cc80c95884"}
Oct 31 01:08:56.482114 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:56.481748    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z" event={"ID":"8c9dee08-2aa4-4c88-bea7-385a01d519bd","Type":"ContainerDied","Data":"9cf65a09f98c85634adb335ecfcf141772e8e71ca11790205c8a4121334aa63e"}
Oct 31 01:08:56.482114 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:56.481774    4327 scope.go:122] "RemoveContainer" containerID="47f2d0934c937efe9e4f16e32e72aecdc84f1840aa5a5ee1f662a3cc80c95884"
Oct 31 01:08:56.492098 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:56.492076    4327 scope.go:122] "RemoveContainer" containerID="47f2d0934c937efe9e4f16e32e72aecdc84f1840aa5a5ee1f662a3cc80c95884"
Oct 31 01:08:56.492381 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:08:56.492357    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f2d0934c937efe9e4f16e32e72aecdc84f1840aa5a5ee1f662a3cc80c95884\": container with ID starting with 47f2d0934c937efe9e4f16e32e72aecdc84f1840aa5a5ee1f662a3cc80c95884 not found: ID does not exist" containerID="47f2d0934c937efe9e4f16e32e72aecdc84f1840aa5a5ee1f662a3cc80c95884"
Oct 31 01:08:56.492438 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:56.492390    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f2d0934c937efe9e4f16e32e72aecdc84f1840aa5a5ee1f662a3cc80c95884"} err="failed to get container status \"47f2d0934c937efe9e4f16e32e72aecdc84f1840aa5a5ee1f662a3cc80c95884\": rpc error: code = NotFound desc = could not find container \"47f2d0934c937efe9e4f16e32e72aecdc84f1840aa5a5ee1f662a3cc80c95884\": container with ID starting with 47f2d0934c937efe9e4f16e32e72aecdc84f1840aa5a5ee1f662a3cc80c95884 not found: ID does not exist"
Oct 31 01:08:56.535726 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:56.535687    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z"]
Oct 31 01:08:56.548314 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:56.548278    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6f8856-64bd4586d8-rz55z"]
Oct 31 01:08:57.922516 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:08:57.922485    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c9dee08-2aa4-4c88-bea7-385a01d519bd" path="/var/lib/kubelet/pods/8c9dee08-2aa4-4c88-bea7-385a01d519bd/volumes"
Oct 31 01:09:02.169946 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:02.169906    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"
Oct 31 01:09:02.183720 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:02.183662    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"
Oct 31 01:09:02.535735 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:02.535638    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"
Oct 31 01:09:03.612846 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:03.612810    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"]
Oct 31 01:09:03.613427 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:03.613399    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4" podUID="46567832-e7e5-42f1-be7b-0bfec97b6b0b" containerName="istio-proxy" containerID="cri-o://44ff5e3660ffd3fcc6879a8028ca89230bf11e97e4e2d2cc893d492b30e1ac89" gracePeriod=30
Oct 31 01:09:05.129110 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.129067    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"]
Oct 31 01:09:05.129993 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.129970    4327 cpu_manager.go:414] "containerMap: removing container" podUID="8c9dee08-2aa4-4c88-bea7-385a01d519bd" containerName="maas-api"
Oct 31 01:09:05.129993 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.129993    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c9dee08-2aa4-4c88-bea7-385a01d519bd" containerName="maas-api"
Oct 31 01:09:05.130178 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.130016    4327 cpu_manager.go:414] "containerMap: removing container" podUID="4ee1484c-003a-4b17-af4a-830ca58a1691" containerName="payload-pre-processing"
Oct 31 01:09:05.130178 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.130024    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ee1484c-003a-4b17-af4a-830ca58a1691" containerName="payload-pre-processing"
Oct 31 01:09:05.130360 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.130345    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c9dee08-2aa4-4c88-bea7-385a01d519bd" containerName="maas-api"
Oct 31 01:09:05.130425 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.130387    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ee1484c-003a-4b17-af4a-830ca58a1691" containerName="payload-pre-processing"
Oct 31 01:09:05.135798 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.135764    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.157882 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.157830    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"]
Oct 31 01:09:05.164041 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.164016    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-73f915-openshift-default-dockercfg-md4jg\""
Oct 31 01:09:05.196203 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.196161    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-data\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.196203 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.196207    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrpg7\" (UniqueName: \"kubernetes.io/projected/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-kube-api-access-rrpg7\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.196439 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.196228    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-podinfo\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.196439 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.196371    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-envoy\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.196439 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.196416    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-workload-certs\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.196612 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.196515    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-workload-socket\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.196654 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.196603    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-credential-socket\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.196654 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.196647    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istiod-ca-cert\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.196881 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.196860    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-token\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.298070 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.298033    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-token\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.298284 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.298103    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-data\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.298284 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.298137    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrpg7\" (UniqueName: \"kubernetes.io/projected/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-kube-api-access-rrpg7\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.298284 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.298169    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-podinfo\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.298284 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.298193    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-envoy\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.298284 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.298224    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-workload-certs\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.298284 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.298267    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-workload-socket\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.298607 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.298322    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-credential-socket\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.298607 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.298370    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istiod-ca-cert\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.298799 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.298757    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-workload-socket\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.298799 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.298786    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-data\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.298971 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.298875    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-credential-socket\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.299033 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.299004    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-workload-certs\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.299273 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.299255    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istiod-ca-cert\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.300670 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.300644    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-envoy\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.300805 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.300741    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-podinfo\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.360626 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.360553    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-token\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.370730 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.370646    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrpg7\" (UniqueName: \"kubernetes.io/projected/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-kube-api-access-rrpg7\") pod \"e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") " pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.454812 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.454453    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:05.777164 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:09:05.777130    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee82474b_1ba2_4ade_8db5_161af9e5f3a4.slice/crio-8ba7bfc640445b2ce652c31431b2cd119c72a40bc2276d802bcd5be860044c14 WatchSource:0}: Error finding container 8ba7bfc640445b2ce652c31431b2cd119c72a40bc2276d802bcd5be860044c14: Status 404 returned error can't find the container with id 8ba7bfc640445b2ce652c31431b2cd119c72a40bc2276d802bcd5be860044c14
Oct 31 01:09:05.778609 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.778584    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"]
Oct 31 01:09:05.780381 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.780344    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:09:05.780472 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.780415    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:09:05.780472 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:05.780446    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:09:06.445177 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:06.445140    4327 patch_prober.go:29] interesting pod/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:09:06.445582 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:06.445200    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4" podUID="46567832-e7e5-42f1-be7b-0bfec97b6b0b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:09:06.563157 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:06.563117    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j" event={"ID":"ee82474b-1ba2-4ade-8db5-161af9e5f3a4","Type":"ContainerStarted","Data":"41695b1410ea58459f9f2133cd6c8f2ea38b45fcefdea9b4eb1b04b8b38e1f75"}
Oct 31 01:09:06.563345 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:06.563180    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j" event={"ID":"ee82474b-1ba2-4ade-8db5-161af9e5f3a4","Type":"ContainerStarted","Data":"8ba7bfc640445b2ce652c31431b2cd119c72a40bc2276d802bcd5be860044c14"}
Oct 31 01:09:06.611192 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:06.611122    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j" podStartSLOduration=1.611107374 podStartE2EDuration="1.611107374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:06.60979204 +0000 UTC m=+3006.831827584" watchObservedRunningTime="2026-10-31 01:09:06.611107374 +0000 UTC m=+3006.833142915"
Oct 31 01:09:07.455396 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:07.455361    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:07.461861 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:07.461835    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:07.570536 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:07.570483    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:07.571487 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:07.571463    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:09:08.895577 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:08.895513    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:09:08.951587 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:08.951545    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istiod-ca-cert\") pod \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") "
Oct 31 01:09:08.951792 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:08.951657    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-podinfo\") pod \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") "
Oct 31 01:09:08.951892 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:08.951797    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-credential-socket\") pod \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") "
Oct 31 01:09:08.951892 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:08.951846    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46567832-e7e5-42f1-be7b-0bfec97b6b0b-kube-api-access-zghtg\" (UniqueName: \"kubernetes.io/projected/46567832-e7e5-42f1-be7b-0bfec97b6b0b-kube-api-access-zghtg\") pod \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") "
Oct 31 01:09:08.951892 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:08.951883    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-workload-certs\") pod \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") "
Oct 31 01:09:08.952052 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:08.951917    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-token\" (UniqueName: \"kubernetes.io/projected/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-token\") pod \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") "
Oct 31 01:09:08.952052 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:08.951974    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-workload-socket\") pod \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") "
Oct 31 01:09:08.952052 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:08.952010    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-data\") pod \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") "
Oct 31 01:09:08.952242 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:08.952047    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istiod-ca-cert" pod "46567832-e7e5-42f1-be7b-0bfec97b6b0b" (UID: "46567832-e7e5-42f1-be7b-0bfec97b6b0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:09:08.952242 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:08.952160    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-envoy\") pod \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\" (UID: \"46567832-e7e5-42f1-be7b-0bfec97b6b0b\") "
Oct 31 01:09:08.952367 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:08.952329    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-workload-certs" pod "46567832-e7e5-42f1-be7b-0bfec97b6b0b" (UID: "46567832-e7e5-42f1-be7b-0bfec97b6b0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:09:08.952551 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:08.952532    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-credential-socket" pod "46567832-e7e5-42f1-be7b-0bfec97b6b0b" (UID: "46567832-e7e5-42f1-be7b-0bfec97b6b0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:09:08.952930 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:08.952843    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:09:08.952930 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:08.952525    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-workload-socket" pod "46567832-e7e5-42f1-be7b-0bfec97b6b0b" (UID: "46567832-e7e5-42f1-be7b-0bfec97b6b0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:09:08.952930 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:08.952869    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:09:08.952930 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:08.952901    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:09:08.953633 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:08.953108    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-data" pod "46567832-e7e5-42f1-be7b-0bfec97b6b0b" (UID: "46567832-e7e5-42f1-be7b-0bfec97b6b0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:09:08.960264 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:08.956942    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46567832-e7e5-42f1-be7b-0bfec97b6b0b-kube-api-access-zghtg" pod "46567832-e7e5-42f1-be7b-0bfec97b6b0b" (UID: "46567832-e7e5-42f1-be7b-0bfec97b6b0b"). InnerVolumeSpecName "kube-api-access-zghtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:09:08.960794 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:08.960756    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-envoy" pod "46567832-e7e5-42f1-be7b-0bfec97b6b0b" (UID: "46567832-e7e5-42f1-be7b-0bfec97b6b0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:09:08.961050 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:08.961025    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-podinfo" pod "46567832-e7e5-42f1-be7b-0bfec97b6b0b" (UID: "46567832-e7e5-42f1-be7b-0bfec97b6b0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:09:08.968203 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:08.968170    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-token" pod "46567832-e7e5-42f1-be7b-0bfec97b6b0b" (UID: "46567832-e7e5-42f1-be7b-0bfec97b6b0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:09:09.054632 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:09.054595    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zghtg\" (UniqueName: \"kubernetes.io/projected/46567832-e7e5-42f1-be7b-0bfec97b6b0b-kube-api-access-zghtg\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:09:09.054632 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:09.054627    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:09:09.054632 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:09.054637    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:09:09.054876 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:09.054647    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:09:09.054876 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:09.054655    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:09:09.054876 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:09.054663    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46567832-e7e5-42f1-be7b-0bfec97b6b0b-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:09:09.586002 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:09.585966    4327 generic.go:356] "Generic (PLEG): container finished" podID="46567832-e7e5-42f1-be7b-0bfec97b6b0b" containerID="44ff5e3660ffd3fcc6879a8028ca89230bf11e97e4e2d2cc893d492b30e1ac89" exitCode=0
Oct 31 01:09:09.586191 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:09.586014    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"
Oct 31 01:09:09.586191 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:09.586059    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4" event={"ID":"46567832-e7e5-42f1-be7b-0bfec97b6b0b","Type":"ContainerDied","Data":"44ff5e3660ffd3fcc6879a8028ca89230bf11e97e4e2d2cc893d492b30e1ac89"}
Oct 31 01:09:09.586191 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:09.586123    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4" event={"ID":"46567832-e7e5-42f1-be7b-0bfec97b6b0b","Type":"ContainerDied","Data":"88394510cd9b75b6ebb3ec952e7f77f5773c03d1f555dca374cb794ba0552835"}
Oct 31 01:09:09.586191 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:09.586151    4327 scope.go:122] "RemoveContainer" containerID="44ff5e3660ffd3fcc6879a8028ca89230bf11e97e4e2d2cc893d492b30e1ac89"
Oct 31 01:09:09.600887 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:09.600868    4327 scope.go:122] "RemoveContainer" containerID="44ff5e3660ffd3fcc6879a8028ca89230bf11e97e4e2d2cc893d492b30e1ac89"
Oct 31 01:09:09.601358 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:09:09.601331    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44ff5e3660ffd3fcc6879a8028ca89230bf11e97e4e2d2cc893d492b30e1ac89\": container with ID starting with 44ff5e3660ffd3fcc6879a8028ca89230bf11e97e4e2d2cc893d492b30e1ac89 not found: ID does not exist" containerID="44ff5e3660ffd3fcc6879a8028ca89230bf11e97e4e2d2cc893d492b30e1ac89"
Oct 31 01:09:09.601466 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:09.601366    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44ff5e3660ffd3fcc6879a8028ca89230bf11e97e4e2d2cc893d492b30e1ac89"} err="failed to get container status \"44ff5e3660ffd3fcc6879a8028ca89230bf11e97e4e2d2cc893d492b30e1ac89\": rpc error: code = NotFound desc = could not find container \"44ff5e3660ffd3fcc6879a8028ca89230bf11e97e4e2d2cc893d492b30e1ac89\": container with ID starting with 44ff5e3660ffd3fcc6879a8028ca89230bf11e97e4e2d2cc893d492b30e1ac89 not found: ID does not exist"
Oct 31 01:09:09.653009 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:09.652972    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"]
Oct 31 01:09:09.672940 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:09.672902    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-rr9z4"]
Oct 31 01:09:09.922787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:09.922752    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46567832-e7e5-42f1-be7b-0bfec97b6b0b" path="/var/lib/kubelet/pods/46567832-e7e5-42f1-be7b-0bfec97b6b0b/volumes"
Oct 31 01:09:10.943229 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:10.943191    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt"]
Oct 31 01:09:10.943923 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:10.943906    4327 cpu_manager.go:414] "containerMap: removing container" podUID="46567832-e7e5-42f1-be7b-0bfec97b6b0b" containerName="istio-proxy"
Oct 31 01:09:10.943995 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:10.943923    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46567832-e7e5-42f1-be7b-0bfec97b6b0b" containerName="istio-proxy"
Oct 31 01:09:10.944092 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:10.944082    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="46567832-e7e5-42f1-be7b-0bfec97b6b0b" containerName="istio-proxy"
Oct 31 01:09:10.949251 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:10.949229    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt"
Oct 31 01:09:10.968358 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:10.968326    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt"]
Oct 31 01:09:10.975214 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:10.975175    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-73f915\""
Oct 31 01:09:10.975411 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:10.975278    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-73f915-dockercfg-92zcl\""
Oct 31 01:09:11.034704 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:11.029696    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78548147-36e3-4c26-84af-51c9c5c593cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt\" (UID: \"78548147-36e3-4c26-84af-51c9c5c593cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt"
Oct 31 01:09:11.034704 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:11.029891    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5t8d\" (UniqueName: \"kubernetes.io/projected/78548147-36e3-4c26-84af-51c9c5c593cd-kube-api-access-k5t8d\") pod \"payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt\" (UID: \"78548147-36e3-4c26-84af-51c9c5c593cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt"
Oct 31 01:09:11.163092 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:11.159714    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78548147-36e3-4c26-84af-51c9c5c593cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt\" (UID: \"78548147-36e3-4c26-84af-51c9c5c593cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt"
Oct 31 01:09:11.163092 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:11.159973    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5t8d\" (UniqueName: \"kubernetes.io/projected/78548147-36e3-4c26-84af-51c9c5c593cd-kube-api-access-k5t8d\") pod \"payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt\" (UID: \"78548147-36e3-4c26-84af-51c9c5c593cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt"
Oct 31 01:09:11.163092 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:11.162959    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78548147-36e3-4c26-84af-51c9c5c593cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt\" (UID: \"78548147-36e3-4c26-84af-51c9c5c593cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt"
Oct 31 01:09:11.183947 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:11.183914    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5t8d\" (UniqueName: \"kubernetes.io/projected/78548147-36e3-4c26-84af-51c9c5c593cd-kube-api-access-k5t8d\") pod \"payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt\" (UID: \"78548147-36e3-4c26-84af-51c9c5c593cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt"
Oct 31 01:09:11.263940 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:11.263852    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt"
Oct 31 01:09:11.580726 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:09:11.580693    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78548147_36e3_4c26_84af_51c9c5c593cd.slice/crio-826aad53a02ba4f0df02090256caa8c918ed8d214ed5845a1836b4737e82166e WatchSource:0}: Error finding container 826aad53a02ba4f0df02090256caa8c918ed8d214ed5845a1836b4737e82166e: Status 404 returned error can't find the container with id 826aad53a02ba4f0df02090256caa8c918ed8d214ed5845a1836b4737e82166e
Oct 31 01:09:11.581885 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:11.581855    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt"]
Oct 31 01:09:11.604580 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:11.604551    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt" event={"ID":"78548147-36e3-4c26-84af-51c9c5c593cd","Type":"ContainerStarted","Data":"826aad53a02ba4f0df02090256caa8c918ed8d214ed5845a1836b4737e82166e"}
Oct 31 01:09:12.633756 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:12.633704    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt" event={"ID":"78548147-36e3-4c26-84af-51c9c5c593cd","Type":"ContainerStarted","Data":"6c8da7621dc7d622894244d9fd0efbd6bf3d49ef0ba507149bbc665128fc16ac"}
Oct 31 01:09:12.634234 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:12.633779    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt"
Oct 31 01:09:12.678239 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:12.678184    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt" podStartSLOduration=2.678170331 podStartE2EDuration="2.678170331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:12.676285005 +0000 UTC m=+3012.898320547" watchObservedRunningTime="2026-10-31 01:09:12.678170331 +0000 UTC m=+3012.900205874"
Oct 31 01:09:15.294391 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:09:15.294356    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:09:23.644182 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:23.644148    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt"
Oct 31 01:09:31.504187 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:31.504151    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"]
Oct 31 01:09:31.509966 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:31.509936    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"
Oct 31 01:09:31.528920 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:31.528883    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d10721\"/\"openshift-service-ca.crt\""
Oct 31 01:09:31.537629 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:31.537594    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d10721\"/\"kube-root-ca.crt\""
Oct 31 01:09:31.541235 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:31.541210    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d10721\"/\"auth-test-model-d10721-kserve-self-signed-certs\""
Oct 31 01:09:31.541389 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:31.541350    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d10721\"/\"default-dockercfg-fzf6b\""
Oct 31 01:09:31.542040 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:31.542015    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"]
Oct 31 01:09:31.690025 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:31.689980    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-dshm\") pod \"auth-test-model-d10721-kserve-5d57899cfc-gkskb\" (UID: \"c65a86d1-1074-4f4c-95db-c7b26378938f\") " pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"
Oct 31 01:09:31.690214 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:31.690104    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-tmp-dir\") pod \"auth-test-model-d10721-kserve-5d57899cfc-gkskb\" (UID: \"c65a86d1-1074-4f4c-95db-c7b26378938f\") " pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"
Oct 31 01:09:31.690214 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:31.690204    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-model-cache\") pod \"auth-test-model-d10721-kserve-5d57899cfc-gkskb\" (UID: \"c65a86d1-1074-4f4c-95db-c7b26378938f\") " pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"
Oct 31 01:09:31.690364 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:31.690344    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qclcp\" (UniqueName: \"kubernetes.io/projected/c65a86d1-1074-4f4c-95db-c7b26378938f-kube-api-access-qclcp\") pod \"auth-test-model-d10721-kserve-5d57899cfc-gkskb\" (UID: \"c65a86d1-1074-4f4c-95db-c7b26378938f\") " pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"
Oct 31 01:09:31.690467 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:31.690447    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c65a86d1-1074-4f4c-95db-c7b26378938f-tls-certs\") pod \"auth-test-model-d10721-kserve-5d57899cfc-gkskb\" (UID: \"c65a86d1-1074-4f4c-95db-c7b26378938f\") " pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"
Oct 31 01:09:31.690566 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:31.690549    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-home\") pod \"auth-test-model-d10721-kserve-5d57899cfc-gkskb\" (UID: \"c65a86d1-1074-4f4c-95db-c7b26378938f\") " pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"
Oct 31 01:09:31.795025 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:31.794853    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-tmp-dir\") pod \"auth-test-model-d10721-kserve-5d57899cfc-gkskb\" (UID: \"c65a86d1-1074-4f4c-95db-c7b26378938f\") " pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"
Oct 31 01:09:31.795025 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:31.794985    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-model-cache\") pod \"auth-test-model-d10721-kserve-5d57899cfc-gkskb\" (UID: \"c65a86d1-1074-4f4c-95db-c7b26378938f\") " pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"
Oct 31 01:09:31.795398 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:31.795077    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qclcp\" (UniqueName: \"kubernetes.io/projected/c65a86d1-1074-4f4c-95db-c7b26378938f-kube-api-access-qclcp\") pod \"auth-test-model-d10721-kserve-5d57899cfc-gkskb\" (UID: \"c65a86d1-1074-4f4c-95db-c7b26378938f\") " pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"
Oct 31 01:09:31.795398 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:31.795146    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c65a86d1-1074-4f4c-95db-c7b26378938f-tls-certs\") pod \"auth-test-model-d10721-kserve-5d57899cfc-gkskb\" (UID: \"c65a86d1-1074-4f4c-95db-c7b26378938f\") " pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"
Oct 31 01:09:31.795398 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:31.795255    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-home\") pod \"auth-test-model-d10721-kserve-5d57899cfc-gkskb\" (UID: \"c65a86d1-1074-4f4c-95db-c7b26378938f\") " pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"
Oct 31 01:09:31.795398 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:31.795314    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-dshm\") pod \"auth-test-model-d10721-kserve-5d57899cfc-gkskb\" (UID: \"c65a86d1-1074-4f4c-95db-c7b26378938f\") " pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"
Oct 31 01:09:31.795398 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:31.795358    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-tmp-dir\") pod \"auth-test-model-d10721-kserve-5d57899cfc-gkskb\" (UID: \"c65a86d1-1074-4f4c-95db-c7b26378938f\") " pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"
Oct 31 01:09:31.796399 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:31.796372    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-home\") pod \"auth-test-model-d10721-kserve-5d57899cfc-gkskb\" (UID: \"c65a86d1-1074-4f4c-95db-c7b26378938f\") " pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"
Oct 31 01:09:31.796719 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:31.796652    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-model-cache\") pod \"auth-test-model-d10721-kserve-5d57899cfc-gkskb\" (UID: \"c65a86d1-1074-4f4c-95db-c7b26378938f\") " pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"
Oct 31 01:09:31.798316 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:31.798291    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-dshm\") pod \"auth-test-model-d10721-kserve-5d57899cfc-gkskb\" (UID: \"c65a86d1-1074-4f4c-95db-c7b26378938f\") " pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"
Oct 31 01:09:31.799468 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:31.799448    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c65a86d1-1074-4f4c-95db-c7b26378938f-tls-certs\") pod \"auth-test-model-d10721-kserve-5d57899cfc-gkskb\" (UID: \"c65a86d1-1074-4f4c-95db-c7b26378938f\") " pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"
Oct 31 01:09:31.819137 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:31.819106    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qclcp\" (UniqueName: \"kubernetes.io/projected/c65a86d1-1074-4f4c-95db-c7b26378938f-kube-api-access-qclcp\") pod \"auth-test-model-d10721-kserve-5d57899cfc-gkskb\" (UID: \"c65a86d1-1074-4f4c-95db-c7b26378938f\") " pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"
Oct 31 01:09:31.827079 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:31.827047    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"
Oct 31 01:09:32.139062 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:09:32.139032    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc65a86d1_1074_4f4c_95db_c7b26378938f.slice/crio-98c219a0a7c0c300314b1ee4a157d799175b4e15c3d81561b6eb30c3286f1ff5 WatchSource:0}: Error finding container 98c219a0a7c0c300314b1ee4a157d799175b4e15c3d81561b6eb30c3286f1ff5: Status 404 returned error can't find the container with id 98c219a0a7c0c300314b1ee4a157d799175b4e15c3d81561b6eb30c3286f1ff5
Oct 31 01:09:32.139921 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:32.139898    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"]
Oct 31 01:09:32.818250 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:32.818211    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb" event={"ID":"c65a86d1-1074-4f4c-95db-c7b26378938f","Type":"ContainerStarted","Data":"3ab4197ab1b276032d060128e1147697bb2ceabc867b7bbca9ec2729f79335dd"}
Oct 31 01:09:32.818657 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:32.818262    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb" event={"ID":"c65a86d1-1074-4f4c-95db-c7b26378938f","Type":"ContainerStarted","Data":"98c219a0a7c0c300314b1ee4a157d799175b4e15c3d81561b6eb30c3286f1ff5"}
Oct 31 01:09:32.827517 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:32.827484    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"
Oct 31 01:09:32.860440 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:32.860385    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb" podStartSLOduration=1.860370066 podStartE2EDuration="1.860370066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:32.8598186 +0000 UTC m=+3033.081854143" watchObservedRunningTime="2026-10-31 01:09:32.860370066 +0000 UTC m=+3033.082405608"
Oct 31 01:09:41.827599 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:41.827558    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"
Oct 31 01:09:41.841290 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:41.841258    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"
Oct 31 01:09:41.923172 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:41.923143    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"
Oct 31 01:09:45.293547 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:09:45.293504    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:09:59.510427 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:59.510392    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"]
Oct 31 01:09:59.516803 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:59.516775    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"
Oct 31 01:09:59.542472 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:59.542445    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"]
Oct 31 01:09:59.548532 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:59.548501    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-ada0c1\"/\"kube-root-ca.crt\""
Oct 31 01:09:59.548736 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:59.548600    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-ada0c1\"/\"openshift-service-ca.crt\""
Oct 31 01:09:59.548736 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:59.548614    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-ada0c1\"/\"default-dockercfg-627sx\""
Oct 31 01:09:59.548736 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:59.548661    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-ada0c1\"/\"auth-test-model-ada0c1-kserve-self-signed-certs\""
Oct 31 01:09:59.629242 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:59.629200    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-dshm\") pod \"auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5\" (UID: \"a283db50-3ca7-494a-8cac-6653efff5251\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"
Oct 31 01:09:59.629242 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:59.629244    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-model-cache\") pod \"auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5\" (UID: \"a283db50-3ca7-494a-8cac-6653efff5251\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"
Oct 31 01:09:59.629463 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:59.629376    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pfbp\" (UniqueName: \"kubernetes.io/projected/a283db50-3ca7-494a-8cac-6653efff5251-kube-api-access-9pfbp\") pod \"auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5\" (UID: \"a283db50-3ca7-494a-8cac-6653efff5251\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"
Oct 31 01:09:59.629463 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:59.629431    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a283db50-3ca7-494a-8cac-6653efff5251-tls-certs\") pod \"auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5\" (UID: \"a283db50-3ca7-494a-8cac-6653efff5251\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"
Oct 31 01:09:59.629560 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:59.629549    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-home\") pod \"auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5\" (UID: \"a283db50-3ca7-494a-8cac-6653efff5251\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"
Oct 31 01:09:59.629623 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:59.629611    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-tmp-dir\") pod \"auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5\" (UID: \"a283db50-3ca7-494a-8cac-6653efff5251\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"
Oct 31 01:09:59.731302 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:59.731262    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-tmp-dir\") pod \"auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5\" (UID: \"a283db50-3ca7-494a-8cac-6653efff5251\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"
Oct 31 01:09:59.731527 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:59.731411    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-dshm\") pod \"auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5\" (UID: \"a283db50-3ca7-494a-8cac-6653efff5251\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"
Oct 31 01:09:59.731527 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:59.731442    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-model-cache\") pod \"auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5\" (UID: \"a283db50-3ca7-494a-8cac-6653efff5251\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"
Oct 31 01:09:59.731527 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:59.731491    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pfbp\" (UniqueName: \"kubernetes.io/projected/a283db50-3ca7-494a-8cac-6653efff5251-kube-api-access-9pfbp\") pod \"auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5\" (UID: \"a283db50-3ca7-494a-8cac-6653efff5251\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"
Oct 31 01:09:59.731746 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:59.731537    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a283db50-3ca7-494a-8cac-6653efff5251-tls-certs\") pod \"auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5\" (UID: \"a283db50-3ca7-494a-8cac-6653efff5251\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"
Oct 31 01:09:59.731746 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:59.731696    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-home\") pod \"auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5\" (UID: \"a283db50-3ca7-494a-8cac-6653efff5251\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"
Oct 31 01:09:59.731858 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:59.731777    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-tmp-dir\") pod \"auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5\" (UID: \"a283db50-3ca7-494a-8cac-6653efff5251\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"
Oct 31 01:09:59.731858 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:59.731843    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-model-cache\") pod \"auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5\" (UID: \"a283db50-3ca7-494a-8cac-6653efff5251\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"
Oct 31 01:09:59.732023 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:59.732003    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-home\") pod \"auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5\" (UID: \"a283db50-3ca7-494a-8cac-6653efff5251\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"
Oct 31 01:09:59.733760 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:59.733736    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-dshm\") pod \"auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5\" (UID: \"a283db50-3ca7-494a-8cac-6653efff5251\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"
Oct 31 01:09:59.734102 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:59.734083    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a283db50-3ca7-494a-8cac-6653efff5251-tls-certs\") pod \"auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5\" (UID: \"a283db50-3ca7-494a-8cac-6653efff5251\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"
Oct 31 01:09:59.784752 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:59.783402    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pfbp\" (UniqueName: \"kubernetes.io/projected/a283db50-3ca7-494a-8cac-6653efff5251-kube-api-access-9pfbp\") pod \"auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5\" (UID: \"a283db50-3ca7-494a-8cac-6653efff5251\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"
Oct 31 01:09:59.846570 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:09:59.846534    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"
Oct 31 01:10:00.154659 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:10:00.154629    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda283db50_3ca7_494a_8cac_6653efff5251.slice/crio-3d9e37edeecd950a31e254f2d2ece3b1b0a4628d71bc028ccc557d8c9c0cc219 WatchSource:0}: Error finding container 3d9e37edeecd950a31e254f2d2ece3b1b0a4628d71bc028ccc557d8c9c0cc219: Status 404 returned error can't find the container with id 3d9e37edeecd950a31e254f2d2ece3b1b0a4628d71bc028ccc557d8c9c0cc219
Oct 31 01:10:00.158805 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:00.158774    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"]
Oct 31 01:10:01.017372 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:01.017333    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5" event={"ID":"a283db50-3ca7-494a-8cac-6653efff5251","Type":"ContainerStarted","Data":"b3ef79d5d73681c7f729b85a4f4bf440242de56392e5e0188c17966b913670e9"}
Oct 31 01:10:01.017372 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:01.017381    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5" event={"ID":"a283db50-3ca7-494a-8cac-6653efff5251","Type":"ContainerStarted","Data":"3d9e37edeecd950a31e254f2d2ece3b1b0a4628d71bc028ccc557d8c9c0cc219"}
Oct 31 01:10:01.065213 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:01.065155    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5" podStartSLOduration=2.065141193 podStartE2EDuration="2.065141193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:01.06132725 +0000 UTC m=+3061.283362793" watchObservedRunningTime="2026-10-31 01:10:01.065141193 +0000 UTC m=+3061.287176735"
Oct 31 01:10:01.847559 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:01.847504    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"
Oct 31 01:10:09.847282 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:09.847240    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"
Oct 31 01:10:09.860535 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:09.860501    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"
Oct 31 01:10:10.137490 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:10.137454    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"
Oct 31 01:10:15.295577 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:10:15.295536    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:10:21.074742 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:21.074707    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt"]
Oct 31 01:10:23.612829 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:23.612790    4327 patch_prober.go:29] interesting pod/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:10:23.613238 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:23.612846    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j" podUID="ee82474b-1ba2-4ade-8db5-161af9e5f3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:10:26.075900 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:26.075856    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt" podUID="78548147-36e3-4c26-84af-51c9c5c593cd" containerName="payload-pre-processing" containerID="cri-o://6c8da7621dc7d622894244d9fd0efbd6bf3d49ef0ba507149bbc665128fc16ac" gracePeriod=25
Oct 31 01:10:26.304975 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:26.304941    4327 generic.go:356] "Generic (PLEG): container finished" podID="78548147-36e3-4c26-84af-51c9c5c593cd" containerID="6c8da7621dc7d622894244d9fd0efbd6bf3d49ef0ba507149bbc665128fc16ac" exitCode=0
Oct 31 01:10:26.305139 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:26.305046    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt" event={"ID":"78548147-36e3-4c26-84af-51c9c5c593cd","Type":"ContainerDied","Data":"6c8da7621dc7d622894244d9fd0efbd6bf3d49ef0ba507149bbc665128fc16ac"}
Oct 31 01:10:26.306711 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:26.306666    4327 generic.go:356] "Generic (PLEG): container finished" podID="ee82474b-1ba2-4ade-8db5-161af9e5f3a4" containerID="41695b1410ea58459f9f2133cd6c8f2ea38b45fcefdea9b4eb1b04b8b38e1f75" exitCode=0
Oct 31 01:10:26.306870 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:26.306810    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j" event={"ID":"ee82474b-1ba2-4ade-8db5-161af9e5f3a4","Type":"ContainerDied","Data":"41695b1410ea58459f9f2133cd6c8f2ea38b45fcefdea9b4eb1b04b8b38e1f75"}
Oct 31 01:10:26.307295 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:26.307276    4327 scope.go:122] "RemoveContainer" containerID="41695b1410ea58459f9f2133cd6c8f2ea38b45fcefdea9b4eb1b04b8b38e1f75"
Oct 31 01:10:26.309184 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:26.309149    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:10:26.309281 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:26.309231    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:10:26.309281 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:26.309278    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:10:26.384587 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:26.384559    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt"
Oct 31 01:10:26.490784 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:26.490744    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78548147-36e3-4c26-84af-51c9c5c593cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78548147-36e3-4c26-84af-51c9c5c593cd-plugins-config-volume\") pod \"78548147-36e3-4c26-84af-51c9c5c593cd\" (UID: \"78548147-36e3-4c26-84af-51c9c5c593cd\") "
Oct 31 01:10:26.490998 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:26.490803    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78548147-36e3-4c26-84af-51c9c5c593cd-kube-api-access-k5t8d\" (UniqueName: \"kubernetes.io/projected/78548147-36e3-4c26-84af-51c9c5c593cd-kube-api-access-k5t8d\") pod \"78548147-36e3-4c26-84af-51c9c5c593cd\" (UID: \"78548147-36e3-4c26-84af-51c9c5c593cd\") "
Oct 31 01:10:26.491764 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:26.491736    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78548147-36e3-4c26-84af-51c9c5c593cd-plugins-config-volume" pod "78548147-36e3-4c26-84af-51c9c5c593cd" (UID: "78548147-36e3-4c26-84af-51c9c5c593cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:10:26.497057 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:26.497032    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78548147-36e3-4c26-84af-51c9c5c593cd-kube-api-access-k5t8d" pod "78548147-36e3-4c26-84af-51c9c5c593cd" (UID: "78548147-36e3-4c26-84af-51c9c5c593cd"). InnerVolumeSpecName "kube-api-access-k5t8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:10:26.593208 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:26.593113    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78548147-36e3-4c26-84af-51c9c5c593cd-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:10:26.593208 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:26.593141    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5t8d\" (UniqueName: \"kubernetes.io/projected/78548147-36e3-4c26-84af-51c9c5c593cd-kube-api-access-k5t8d\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:10:27.314522 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:27.314486    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt"
Oct 31 01:10:27.314522 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:27.314516    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt" event={"ID":"78548147-36e3-4c26-84af-51c9c5c593cd","Type":"ContainerDied","Data":"826aad53a02ba4f0df02090256caa8c918ed8d214ed5845a1836b4737e82166e"}
Oct 31 01:10:27.315086 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:27.314584    4327 scope.go:122] "RemoveContainer" containerID="6c8da7621dc7d622894244d9fd0efbd6bf3d49ef0ba507149bbc665128fc16ac"
Oct 31 01:10:27.343704 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:27.337240    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j" event={"ID":"ee82474b-1ba2-4ade-8db5-161af9e5f3a4","Type":"ContainerStarted","Data":"d3663e71438b0464804dc5e7b9c7be074cdf36a8b6d5c4a25bf85b623763ec5f"}
Oct 31 01:10:27.448125 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:27.448081    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt"]
Oct 31 01:10:27.451771 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:27.451744    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-d658955ff-lphgt"]
Oct 31 01:10:27.455732 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:27.455712    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:10:27.455915 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:27.455896    4327 patch_prober.go:29] interesting pod/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 31 01:10:27.455972 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:27.455938    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j" podUID="ee82474b-1ba2-4ade-8db5-161af9e5f3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 31 01:10:27.950969 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:27.950939    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78548147-36e3-4c26-84af-51c9c5c593cd" path="/var/lib/kubelet/pods/78548147-36e3-4c26-84af-51c9c5c593cd/volumes"
Oct 31 01:10:28.456119 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:28.456080    4327 patch_prober.go:29] interesting pod/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 31 01:10:28.456536 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:28.456130    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j" podUID="ee82474b-1ba2-4ade-8db5-161af9e5f3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 31 01:10:29.456114 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:29.456075    4327 patch_prober.go:29] interesting pod/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 31 01:10:29.456330 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:29.456153    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j" podUID="ee82474b-1ba2-4ade-8db5-161af9e5f3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 31 01:10:30.456191 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:30.456156    4327 patch_prober.go:29] interesting pod/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 31 01:10:30.456514 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:30.456227    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j" podUID="ee82474b-1ba2-4ade-8db5-161af9e5f3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 31 01:10:31.455407 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:31.455372    4327 patch_prober.go:29] interesting pod/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 31 01:10:31.455596 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:31.455430    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j" podUID="ee82474b-1ba2-4ade-8db5-161af9e5f3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 31 01:10:32.455524 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:32.455486    4327 patch_prober.go:29] interesting pod/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 31 01:10:32.455972 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:32.455548    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j" podUID="ee82474b-1ba2-4ade-8db5-161af9e5f3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 31 01:10:33.460067 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:33.460025    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:10:33.460489 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:33.460333    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:10:33.461109 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:10:33.461092    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:10:45.294256 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:10:45.294217    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:11:14.835914 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:14.835876    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54"]
Oct 31 01:11:14.836662 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:14.836644    4327 cpu_manager.go:414] "containerMap: removing container" podUID="78548147-36e3-4c26-84af-51c9c5c593cd" containerName="payload-pre-processing"
Oct 31 01:11:14.836751 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:14.836663    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78548147-36e3-4c26-84af-51c9c5c593cd" containerName="payload-pre-processing"
Oct 31 01:11:14.836791 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:14.836784    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="78548147-36e3-4c26-84af-51c9c5c593cd" containerName="payload-pre-processing"
Oct 31 01:11:14.841808 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:14.841783    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54"
Oct 31 01:11:14.859028 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:14.858998    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54"]
Oct 31 01:11:14.860374 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:14.860349    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-75ead8-dockercfg-jh67k\""
Oct 31 01:11:14.867257 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:14.867222    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-75ead8\""
Oct 31 01:11:14.910372 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:14.910333    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75zcw\" (UniqueName: \"kubernetes.io/projected/100a0f8a-a9c4-458c-ba81-e61970f18cdf-kube-api-access-75zcw\") pod \"payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54\" (UID: \"100a0f8a-a9c4-458c-ba81-e61970f18cdf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54"
Oct 31 01:11:14.910575 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:14.910402    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100a0f8a-a9c4-458c-ba81-e61970f18cdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54\" (UID: \"100a0f8a-a9c4-458c-ba81-e61970f18cdf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54"
Oct 31 01:11:15.076706 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:15.074139    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm"]
Oct 31 01:11:15.079390 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:15.079285    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75zcw\" (UniqueName: \"kubernetes.io/projected/100a0f8a-a9c4-458c-ba81-e61970f18cdf-kube-api-access-75zcw\") pod \"payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54\" (UID: \"100a0f8a-a9c4-458c-ba81-e61970f18cdf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54"
Oct 31 01:11:15.083821 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:15.083787    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100a0f8a-a9c4-458c-ba81-e61970f18cdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54\" (UID: \"100a0f8a-a9c4-458c-ba81-e61970f18cdf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54"
Oct 31 01:11:15.090812 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:15.090623    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100a0f8a-a9c4-458c-ba81-e61970f18cdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54\" (UID: \"100a0f8a-a9c4-458c-ba81-e61970f18cdf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54"
Oct 31 01:11:15.117276 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:15.116756    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75zcw\" (UniqueName: \"kubernetes.io/projected/100a0f8a-a9c4-458c-ba81-e61970f18cdf-kube-api-access-75zcw\") pod \"payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54\" (UID: \"100a0f8a-a9c4-458c-ba81-e61970f18cdf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54"
Oct 31 01:11:15.125020 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:15.124988    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm"]
Oct 31 01:11:15.125244 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:15.125135    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm"
Oct 31 01:11:15.156940 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:15.156909    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54"
Oct 31 01:11:15.343871 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:15.343501    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aae3bb8-04d9-40e1-bcea-476334fa6363-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm\" (UID: \"0aae3bb8-04d9-40e1-bcea-476334fa6363\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm"
Oct 31 01:11:15.343871 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:15.343788    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwtm8\" (UniqueName: \"kubernetes.io/projected/0aae3bb8-04d9-40e1-bcea-476334fa6363-kube-api-access-kwtm8\") pod \"payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm\" (UID: \"0aae3bb8-04d9-40e1-bcea-476334fa6363\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm"
Oct 31 01:11:15.456385 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:15.456326    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aae3bb8-04d9-40e1-bcea-476334fa6363-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm\" (UID: \"0aae3bb8-04d9-40e1-bcea-476334fa6363\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm"
Oct 31 01:11:15.456586 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:15.456562    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwtm8\" (UniqueName: \"kubernetes.io/projected/0aae3bb8-04d9-40e1-bcea-476334fa6363-kube-api-access-kwtm8\") pod \"payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm\" (UID: \"0aae3bb8-04d9-40e1-bcea-476334fa6363\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm"
Oct 31 01:11:15.458155 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:15.458125    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aae3bb8-04d9-40e1-bcea-476334fa6363-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm\" (UID: \"0aae3bb8-04d9-40e1-bcea-476334fa6363\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm"
Oct 31 01:11:15.461905 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:11:15.461869    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:11:15.483717 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:15.481939    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwtm8\" (UniqueName: \"kubernetes.io/projected/0aae3bb8-04d9-40e1-bcea-476334fa6363-kube-api-access-kwtm8\") pod \"payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm\" (UID: \"0aae3bb8-04d9-40e1-bcea-476334fa6363\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm"
Oct 31 01:11:15.498417 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:11:15.498382    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100a0f8a_a9c4_458c_ba81_e61970f18cdf.slice/crio-fda7958e90eeac57cd599a75b0385cac3f74d4b7d6e0e93febf3e2dfa847ebc5 WatchSource:0}: Error finding container fda7958e90eeac57cd599a75b0385cac3f74d4b7d6e0e93febf3e2dfa847ebc5: Status 404 returned error can't find the container with id fda7958e90eeac57cd599a75b0385cac3f74d4b7d6e0e93febf3e2dfa847ebc5
Oct 31 01:11:15.499950 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:15.499924    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54"]
Oct 31 01:11:15.700021 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:15.699978    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54" event={"ID":"100a0f8a-a9c4-458c-ba81-e61970f18cdf","Type":"ContainerStarted","Data":"f5bdd7017942890bbdb7dd5e3b774a18c10c3456e70077a2e3f543880f7d3186"}
Oct 31 01:11:15.700217 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:15.700044    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54" event={"ID":"100a0f8a-a9c4-458c-ba81-e61970f18cdf","Type":"ContainerStarted","Data":"fda7958e90eeac57cd599a75b0385cac3f74d4b7d6e0e93febf3e2dfa847ebc5"}
Oct 31 01:11:15.700217 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:15.700182    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54"
Oct 31 01:11:15.741711 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:15.741273    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54" podStartSLOduration=1.741253769 podStartE2EDuration="1.741253769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:15.740027344 +0000 UTC m=+3135.962062887" watchObservedRunningTime="2026-10-31 01:11:15.741253769 +0000 UTC m=+3135.963289312"
Oct 31 01:11:15.760705 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:15.748497    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm"
Oct 31 01:11:16.083341 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:11:16.083313    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aae3bb8_04d9_40e1_bcea_476334fa6363.slice/crio-6a9a805e589145bdb7a40ce66fd32d6d290df086da8ef47b3836c8c330691c70 WatchSource:0}: Error finding container 6a9a805e589145bdb7a40ce66fd32d6d290df086da8ef47b3836c8c330691c70: Status 404 returned error can't find the container with id 6a9a805e589145bdb7a40ce66fd32d6d290df086da8ef47b3836c8c330691c70
Oct 31 01:11:16.083870 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:16.083846    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm"]
Oct 31 01:11:16.710085 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:16.710039    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm" event={"ID":"0aae3bb8-04d9-40e1-bcea-476334fa6363","Type":"ContainerStarted","Data":"86679373ac565aec247c997de393c568f0e02d4e6ddb961cdfb313868342289f"}
Oct 31 01:11:16.710276 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:16.710117    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm"
Oct 31 01:11:16.710276 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:16.710138    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm" event={"ID":"0aae3bb8-04d9-40e1-bcea-476334fa6363","Type":"ContainerStarted","Data":"6a9a805e589145bdb7a40ce66fd32d6d290df086da8ef47b3836c8c330691c70"}
Oct 31 01:11:16.782441 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:16.782384    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm" podStartSLOduration=2.782367211 podStartE2EDuration="2.782367211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:16.781447873 +0000 UTC m=+3137.003483415" watchObservedRunningTime="2026-10-31 01:11:16.782367211 +0000 UTC m=+3137.004402753"
Oct 31 01:11:26.713884 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:26.713851    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54"
Oct 31 01:11:27.761321 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:27.761290    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm"
Oct 31 01:11:39.787004 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:39.786961    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"]
Oct 31 01:11:45.293837 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:11:45.293792    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:11:54.811754 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:54.811699    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w" podUID="ef6850b2-d382-4728-9f01-72e9639adefc" containerName="main" containerID="cri-o://9512722b45aa5f454a7bfbf95c7664cb5c24fc729b9c6d0464bcd423b50705bb" gracePeriod=45
Oct 31 01:11:54.874013 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:54.873972    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w" podUID="ef6850b2-d382-4728-9f01-72e9639adefc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.191:8000/ready\": dial tcp 10.133.0.191:8000: connect: connection refused"
Oct 31 01:11:55.001370 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:55.001136    4327 generic.go:356] "Generic (PLEG): container finished" podID="ef6850b2-d382-4728-9f01-72e9639adefc" containerID="9512722b45aa5f454a7bfbf95c7664cb5c24fc729b9c6d0464bcd423b50705bb" exitCode=0
Oct 31 01:11:55.001370 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:55.001352    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w" event={"ID":"ef6850b2-d382-4728-9f01-72e9639adefc","Type":"ContainerDied","Data":"9512722b45aa5f454a7bfbf95c7664cb5c24fc729b9c6d0464bcd423b50705bb"}
Oct 31 01:11:55.095523 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:55.095496    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"
Oct 31 01:11:55.262940 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:55.262904    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-home\" (UniqueName: \"kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-home\") pod \"ef6850b2-d382-4728-9f01-72e9639adefc\" (UID: \"ef6850b2-d382-4728-9f01-72e9639adefc\") "
Oct 31 01:11:55.263127 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:55.262961    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-model-cache\") pod \"ef6850b2-d382-4728-9f01-72e9639adefc\" (UID: \"ef6850b2-d382-4728-9f01-72e9639adefc\") "
Oct 31 01:11:55.263127 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:55.263117    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef6850b2-d382-4728-9f01-72e9639adefc-tls-certs\" (UniqueName: \"kubernetes.io/secret/ef6850b2-d382-4728-9f01-72e9639adefc-tls-certs\") pod \"ef6850b2-d382-4728-9f01-72e9639adefc\" (UID: \"ef6850b2-d382-4728-9f01-72e9639adefc\") "
Oct 31 01:11:55.263230 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:55.263138    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-tmp-dir\") pod \"ef6850b2-d382-4728-9f01-72e9639adefc\" (UID: \"ef6850b2-d382-4728-9f01-72e9639adefc\") "
Oct 31 01:11:55.263230 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:55.263186    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6850b2-d382-4728-9f01-72e9639adefc-kube-api-access-xsd22\" (UniqueName: \"kubernetes.io/projected/ef6850b2-d382-4728-9f01-72e9639adefc-kube-api-access-xsd22\") pod \"ef6850b2-d382-4728-9f01-72e9639adefc\" (UID: \"ef6850b2-d382-4728-9f01-72e9639adefc\") "
Oct 31 01:11:55.263230 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:55.263183    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-home" pod "ef6850b2-d382-4728-9f01-72e9639adefc" (UID: "ef6850b2-d382-4728-9f01-72e9639adefc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:11:55.263407 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:55.263228    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-model-cache" pod "ef6850b2-d382-4728-9f01-72e9639adefc" (UID: "ef6850b2-d382-4728-9f01-72e9639adefc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:11:55.263407 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:55.263291    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-dshm\") pod \"ef6850b2-d382-4728-9f01-72e9639adefc\" (UID: \"ef6850b2-d382-4728-9f01-72e9639adefc\") "
Oct 31 01:11:55.263517 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:55.263414    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-tmp-dir" pod "ef6850b2-d382-4728-9f01-72e9639adefc" (UID: "ef6850b2-d382-4728-9f01-72e9639adefc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:11:55.263962 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:55.263930    4327 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-model-cache\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:11:55.263962 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:55.263961    4327 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-tmp-dir\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:11:55.264147 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:55.263971    4327 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-home\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:11:55.265587 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:55.265560    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef6850b2-d382-4728-9f01-72e9639adefc-tls-certs" pod "ef6850b2-d382-4728-9f01-72e9639adefc" (UID: "ef6850b2-d382-4728-9f01-72e9639adefc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:11:55.265725 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:55.265636    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-dshm" pod "ef6850b2-d382-4728-9f01-72e9639adefc" (UID: "ef6850b2-d382-4728-9f01-72e9639adefc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:11:55.265725 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:55.265649    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6850b2-d382-4728-9f01-72e9639adefc-kube-api-access-xsd22" pod "ef6850b2-d382-4728-9f01-72e9639adefc" (UID: "ef6850b2-d382-4728-9f01-72e9639adefc"). InnerVolumeSpecName "kube-api-access-xsd22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:11:55.365642 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:55.365610    4327 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef6850b2-d382-4728-9f01-72e9639adefc-tls-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:11:55.365642 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:55.365639    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsd22\" (UniqueName: \"kubernetes.io/projected/ef6850b2-d382-4728-9f01-72e9639adefc-kube-api-access-xsd22\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:11:55.365642 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:55.365649    4327 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef6850b2-d382-4728-9f01-72e9639adefc-dshm\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:11:56.010180 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:56.010148    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"
Oct 31 01:11:56.010623 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:56.010150    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w" event={"ID":"ef6850b2-d382-4728-9f01-72e9639adefc","Type":"ContainerDied","Data":"37e05e63f9997a514b861bd8fc6ee767c092fdab63fcaa5e6045edf674a3bfce"}
Oct 31 01:11:56.010623 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:56.010280    4327 scope.go:122] "RemoveContainer" containerID="9512722b45aa5f454a7bfbf95c7664cb5c24fc729b9c6d0464bcd423b50705bb"
Oct 31 01:11:56.064855 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:56.064815    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"]
Oct 31 01:11:56.076616 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:56.076583    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-36af31/auth-test-model-36af31-kserve-847b468c57-bxs2w"]
Oct 31 01:11:57.923430 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:11:57.923398    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef6850b2-d382-4728-9f01-72e9639adefc" path="/var/lib/kubelet/pods/ef6850b2-d382-4728-9f01-72e9639adefc/volumes"
Oct 31 01:12:00.519365 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:00.519324    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"]
Oct 31 01:12:15.294309 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:12:15.294225    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:12:15.551171 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:15.551070    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w" podUID="9cc1b090-45a2-4aa0-a778-a4b18f8e197c" containerName="main" containerID="cri-o://777cfaf996e8915907fe2fbe52676816d1135b132c6dcc17e72a72eae4c8596c" gracePeriod=45
Oct 31 01:12:15.830704 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:15.830649    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"
Oct 31 01:12:15.995828 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:15.995778    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-kube-api-access-8q6nc\" (UniqueName: \"kubernetes.io/projected/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-kube-api-access-8q6nc\") pod \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\" (UID: \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\") "
Oct 31 01:12:15.996044 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:15.995893    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-dshm\") pod \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\" (UID: \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\") "
Oct 31 01:12:15.996044 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:15.995947    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-tls-certs\" (UniqueName: \"kubernetes.io/secret/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-tls-certs\") pod \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\" (UID: \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\") "
Oct 31 01:12:15.996225 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:15.996076    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-model-cache\") pod \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\" (UID: \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\") "
Oct 31 01:12:15.996225 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:15.996123    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-tmp-dir\") pod \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\" (UID: \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\") "
Oct 31 01:12:15.996225 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:15.996187    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-home\" (UniqueName: \"kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-home\") pod \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\" (UID: \"9cc1b090-45a2-4aa0-a778-a4b18f8e197c\") "
Oct 31 01:12:15.996555 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:15.996516    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-tmp-dir" pod "9cc1b090-45a2-4aa0-a778-a4b18f8e197c" (UID: "9cc1b090-45a2-4aa0-a778-a4b18f8e197c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:12:15.996720 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:15.996624    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-model-cache" pod "9cc1b090-45a2-4aa0-a778-a4b18f8e197c" (UID: "9cc1b090-45a2-4aa0-a778-a4b18f8e197c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:12:15.996720 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:15.996644    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-home" pod "9cc1b090-45a2-4aa0-a778-a4b18f8e197c" (UID: "9cc1b090-45a2-4aa0-a778-a4b18f8e197c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:12:15.997009 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:15.996987    4327 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-model-cache\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:12:15.997051 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:15.997018    4327 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-tmp-dir\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:12:15.997051 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:15.997035    4327 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-home\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:12:15.998354 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:15.998329    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-tls-certs" pod "9cc1b090-45a2-4aa0-a778-a4b18f8e197c" (UID: "9cc1b090-45a2-4aa0-a778-a4b18f8e197c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:12:15.998354 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:15.998339    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-dshm" pod "9cc1b090-45a2-4aa0-a778-a4b18f8e197c" (UID: "9cc1b090-45a2-4aa0-a778-a4b18f8e197c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:12:15.998532 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:15.998515    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-kube-api-access-8q6nc" pod "9cc1b090-45a2-4aa0-a778-a4b18f8e197c" (UID: "9cc1b090-45a2-4aa0-a778-a4b18f8e197c"). InnerVolumeSpecName "kube-api-access-8q6nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:12:16.098489 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:16.098402    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q6nc\" (UniqueName: \"kubernetes.io/projected/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-kube-api-access-8q6nc\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:12:16.098489 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:16.098437    4327 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-dshm\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:12:16.098489 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:16.098448    4327 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cc1b090-45a2-4aa0-a778-a4b18f8e197c-tls-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:12:16.213491 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:16.213449    4327 generic.go:356] "Generic (PLEG): container finished" podID="9cc1b090-45a2-4aa0-a778-a4b18f8e197c" containerID="777cfaf996e8915907fe2fbe52676816d1135b132c6dcc17e72a72eae4c8596c" exitCode=0
Oct 31 01:12:16.213764 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:16.213507    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"
Oct 31 01:12:16.213764 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:16.213632    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w" event={"ID":"9cc1b090-45a2-4aa0-a778-a4b18f8e197c","Type":"ContainerDied","Data":"777cfaf996e8915907fe2fbe52676816d1135b132c6dcc17e72a72eae4c8596c"}
Oct 31 01:12:16.213764 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:16.213673    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w" event={"ID":"9cc1b090-45a2-4aa0-a778-a4b18f8e197c","Type":"ContainerDied","Data":"171eecb00176b01acd42e5a3a6cfecc02bdb26219bc6486514263b551a9ec78c"}
Oct 31 01:12:16.213764 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:16.213741    4327 scope.go:122] "RemoveContainer" containerID="777cfaf996e8915907fe2fbe52676816d1135b132c6dcc17e72a72eae4c8596c"
Oct 31 01:12:16.225993 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:16.225971    4327 scope.go:122] "RemoveContainer" containerID="777cfaf996e8915907fe2fbe52676816d1135b132c6dcc17e72a72eae4c8596c"
Oct 31 01:12:16.226284 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:12:16.226260    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"777cfaf996e8915907fe2fbe52676816d1135b132c6dcc17e72a72eae4c8596c\": container with ID starting with 777cfaf996e8915907fe2fbe52676816d1135b132c6dcc17e72a72eae4c8596c not found: ID does not exist" containerID="777cfaf996e8915907fe2fbe52676816d1135b132c6dcc17e72a72eae4c8596c"
Oct 31 01:12:16.226363 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:16.226297    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"777cfaf996e8915907fe2fbe52676816d1135b132c6dcc17e72a72eae4c8596c"} err="failed to get container status \"777cfaf996e8915907fe2fbe52676816d1135b132c6dcc17e72a72eae4c8596c\": rpc error: code = NotFound desc = could not find container \"777cfaf996e8915907fe2fbe52676816d1135b132c6dcc17e72a72eae4c8596c\": container with ID starting with 777cfaf996e8915907fe2fbe52676816d1135b132c6dcc17e72a72eae4c8596c not found: ID does not exist"
Oct 31 01:12:16.269625 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:16.269583    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"]
Oct 31 01:12:16.281829 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:16.281789    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f2bdf3/auth-test-model-f2bdf3-kserve-647bbf4457-7gp9w"]
Oct 31 01:12:17.923307 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:17.923269    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cc1b090-45a2-4aa0-a778-a4b18f8e197c" path="/var/lib/kubelet/pods/9cc1b090-45a2-4aa0-a778-a4b18f8e197c/volumes"
Oct 31 01:12:28.564788 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:28.564748    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z"]
Oct 31 01:12:28.565584 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:28.565566    4327 cpu_manager.go:414] "containerMap: removing container" podUID="ef6850b2-d382-4728-9f01-72e9639adefc" containerName="main"
Oct 31 01:12:28.565634 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:28.565585    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6850b2-d382-4728-9f01-72e9639adefc" containerName="main"
Oct 31 01:12:28.565634 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:28.565601    4327 cpu_manager.go:414] "containerMap: removing container" podUID="9cc1b090-45a2-4aa0-a778-a4b18f8e197c" containerName="main"
Oct 31 01:12:28.565634 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:28.565607    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cc1b090-45a2-4aa0-a778-a4b18f8e197c" containerName="main"
Oct 31 01:12:28.565784 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:28.565743    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6850b2-d382-4728-9f01-72e9639adefc" containerName="main"
Oct 31 01:12:28.565784 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:28.565754    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cc1b090-45a2-4aa0-a778-a4b18f8e197c" containerName="main"
Oct 31 01:12:28.570915 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:28.570895    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z"
Oct 31 01:12:28.588409 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:28.588385    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z"]
Oct 31 01:12:28.596708 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:28.596671    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e41918-dockercfg-x82js\""
Oct 31 01:12:28.596885 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:28.596780    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e41918\""
Oct 31 01:12:28.659795 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:28.655616    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6swbw\" (UniqueName: \"kubernetes.io/projected/9f274877-9424-47bc-9e10-5df2c2455a4b-kube-api-access-6swbw\") pod \"payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z\" (UID: \"9f274877-9424-47bc-9e10-5df2c2455a4b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z"
Oct 31 01:12:28.673703 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:28.661910    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f274877-9424-47bc-9e10-5df2c2455a4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z\" (UID: \"9f274877-9424-47bc-9e10-5df2c2455a4b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z"
Oct 31 01:12:28.753729 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:28.752766    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d"]
Oct 31 01:12:28.757625 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:28.757593    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d"
Oct 31 01:12:28.759135 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:28.759103    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d"]
Oct 31 01:12:28.791619 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:28.791569    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6swbw\" (UniqueName: \"kubernetes.io/projected/9f274877-9424-47bc-9e10-5df2c2455a4b-kube-api-access-6swbw\") pod \"payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z\" (UID: \"9f274877-9424-47bc-9e10-5df2c2455a4b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z"
Oct 31 01:12:28.791885 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:28.791811    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f274877-9424-47bc-9e10-5df2c2455a4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z\" (UID: \"9f274877-9424-47bc-9e10-5df2c2455a4b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z"
Oct 31 01:12:28.792855 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:28.792802    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f274877-9424-47bc-9e10-5df2c2455a4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z\" (UID: \"9f274877-9424-47bc-9e10-5df2c2455a4b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z"
Oct 31 01:12:28.816238 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:28.816172    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6swbw\" (UniqueName: \"kubernetes.io/projected/9f274877-9424-47bc-9e10-5df2c2455a4b-kube-api-access-6swbw\") pod \"payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z\" (UID: \"9f274877-9424-47bc-9e10-5df2c2455a4b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z"
Oct 31 01:12:28.884994 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:28.884961    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z"
Oct 31 01:12:28.892738 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:28.892712    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bzk9\" (UniqueName: \"kubernetes.io/projected/6d7550dd-2f21-4f11-bf31-ea0979421f9e-kube-api-access-2bzk9\") pod \"payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d\" (UID: \"6d7550dd-2f21-4f11-bf31-ea0979421f9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d"
Oct 31 01:12:28.892915 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:28.892889    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d7550dd-2f21-4f11-bf31-ea0979421f9e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d\" (UID: \"6d7550dd-2f21-4f11-bf31-ea0979421f9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d"
Oct 31 01:12:28.994543 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:28.994466    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d7550dd-2f21-4f11-bf31-ea0979421f9e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d\" (UID: \"6d7550dd-2f21-4f11-bf31-ea0979421f9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d"
Oct 31 01:12:28.994746 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:28.994704    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bzk9\" (UniqueName: \"kubernetes.io/projected/6d7550dd-2f21-4f11-bf31-ea0979421f9e-kube-api-access-2bzk9\") pod \"payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d\" (UID: \"6d7550dd-2f21-4f11-bf31-ea0979421f9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d"
Oct 31 01:12:28.995929 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:28.995898    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d7550dd-2f21-4f11-bf31-ea0979421f9e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d\" (UID: \"6d7550dd-2f21-4f11-bf31-ea0979421f9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d"
Oct 31 01:12:29.034616 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:29.034570    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bzk9\" (UniqueName: \"kubernetes.io/projected/6d7550dd-2f21-4f11-bf31-ea0979421f9e-kube-api-access-2bzk9\") pod \"payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d\" (UID: \"6d7550dd-2f21-4f11-bf31-ea0979421f9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d"
Oct 31 01:12:29.077868 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:29.077792    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d"
Oct 31 01:12:29.195477 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:29.195446    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z"]
Oct 31 01:12:29.325145 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:29.325102    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z" event={"ID":"9f274877-9424-47bc-9e10-5df2c2455a4b","Type":"ContainerStarted","Data":"41ca7d2600aeeccf36fd61d010f29e250febd49b516c4e5d2a01f251bcb64220"}
Oct 31 01:12:29.325325 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:29.325179    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z"
Oct 31 01:12:29.325325 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:29.325203    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z" event={"ID":"9f274877-9424-47bc-9e10-5df2c2455a4b","Type":"ContainerStarted","Data":"884dc118f70f2ca971303aa12546996898e77a8d8e301770e81037ab4891c9d6"}
Oct 31 01:12:29.371488 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:29.371429    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z" podStartSLOduration=1.371415016 podStartE2EDuration="1.371415016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:29.367208144 +0000 UTC m=+3209.589243693" watchObservedRunningTime="2026-10-31 01:12:29.371415016 +0000 UTC m=+3209.593450557"
Oct 31 01:12:29.395753 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:12:29.395713    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d7550dd_2f21_4f11_bf31_ea0979421f9e.slice/crio-ab1a5da2babb45ec8191d2e74cf856414315a4de4a243b06fefc0f9c638d98a2 WatchSource:0}: Error finding container ab1a5da2babb45ec8191d2e74cf856414315a4de4a243b06fefc0f9c638d98a2: Status 404 returned error can't find the container with id ab1a5da2babb45ec8191d2e74cf856414315a4de4a243b06fefc0f9c638d98a2
Oct 31 01:12:29.397334 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:29.397305    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d"]
Oct 31 01:12:29.710127 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:29.710045    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm"]
Oct 31 01:12:29.739844 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:29.739797    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54"]
Oct 31 01:12:30.396775 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:30.385952    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d" event={"ID":"6d7550dd-2f21-4f11-bf31-ea0979421f9e","Type":"ContainerStarted","Data":"6fd9d7b18b422921abf0ab26e7af42c1ac8e9535ba476e8d56f1adf7eb2b3ad7"}
Oct 31 01:12:30.396775 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:30.386018    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d" event={"ID":"6d7550dd-2f21-4f11-bf31-ea0979421f9e","Type":"ContainerStarted","Data":"ab1a5da2babb45ec8191d2e74cf856414315a4de4a243b06fefc0f9c638d98a2"}
Oct 31 01:12:30.396775 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:30.386075    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d"
Oct 31 01:12:30.435290 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:30.435230    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d" podStartSLOduration=2.435205313 podStartE2EDuration="2.435205313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:30.432497811 +0000 UTC m=+3210.654533352" watchObservedRunningTime="2026-10-31 01:12:30.435205313 +0000 UTC m=+3210.657240857"
Oct 31 01:12:34.711078 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:34.711010    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm" podUID="0aae3bb8-04d9-40e1-bcea-476334fa6363" containerName="payload-processing" containerID="cri-o://86679373ac565aec247c997de393c568f0e02d4e6ddb961cdfb313868342289f" gracePeriod=25
Oct 31 01:12:34.740785 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:34.740726    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54" podUID="100a0f8a-a9c4-458c-ba81-e61970f18cdf" containerName="payload-pre-processing" containerID="cri-o://f5bdd7017942890bbdb7dd5e3b774a18c10c3456e70077a2e3f543880f7d3186" gracePeriod=25
Oct 31 01:12:35.116583 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.116556    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm"
Oct 31 01:12:35.171391 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.171239    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54"
Oct 31 01:12:35.203223 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.203187    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aae3bb8-04d9-40e1-bcea-476334fa6363-kube-api-access-kwtm8\" (UniqueName: \"kubernetes.io/projected/0aae3bb8-04d9-40e1-bcea-476334fa6363-kube-api-access-kwtm8\") pod \"0aae3bb8-04d9-40e1-bcea-476334fa6363\" (UID: \"0aae3bb8-04d9-40e1-bcea-476334fa6363\") "
Oct 31 01:12:35.203223 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.203222    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aae3bb8-04d9-40e1-bcea-476334fa6363-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aae3bb8-04d9-40e1-bcea-476334fa6363-plugins-config-volume\") pod \"0aae3bb8-04d9-40e1-bcea-476334fa6363\" (UID: \"0aae3bb8-04d9-40e1-bcea-476334fa6363\") "
Oct 31 01:12:35.203640 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.203616    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aae3bb8-04d9-40e1-bcea-476334fa6363-plugins-config-volume" pod "0aae3bb8-04d9-40e1-bcea-476334fa6363" (UID: "0aae3bb8-04d9-40e1-bcea-476334fa6363"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:12:35.203927 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.203902    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aae3bb8-04d9-40e1-bcea-476334fa6363-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:12:35.205329 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.205308    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aae3bb8-04d9-40e1-bcea-476334fa6363-kube-api-access-kwtm8" pod "0aae3bb8-04d9-40e1-bcea-476334fa6363" (UID: "0aae3bb8-04d9-40e1-bcea-476334fa6363"). InnerVolumeSpecName "kube-api-access-kwtm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:12:35.305283 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.305199    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/100a0f8a-a9c4-458c-ba81-e61970f18cdf-kube-api-access-75zcw\" (UniqueName: \"kubernetes.io/projected/100a0f8a-a9c4-458c-ba81-e61970f18cdf-kube-api-access-75zcw\") pod \"100a0f8a-a9c4-458c-ba81-e61970f18cdf\" (UID: \"100a0f8a-a9c4-458c-ba81-e61970f18cdf\") "
Oct 31 01:12:35.305283 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.305236    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/100a0f8a-a9c4-458c-ba81-e61970f18cdf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100a0f8a-a9c4-458c-ba81-e61970f18cdf-plugins-config-volume\") pod \"100a0f8a-a9c4-458c-ba81-e61970f18cdf\" (UID: \"100a0f8a-a9c4-458c-ba81-e61970f18cdf\") "
Oct 31 01:12:35.305667 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.305642    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/100a0f8a-a9c4-458c-ba81-e61970f18cdf-plugins-config-volume" pod "100a0f8a-a9c4-458c-ba81-e61970f18cdf" (UID: "100a0f8a-a9c4-458c-ba81-e61970f18cdf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:12:35.305883 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.305865    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100a0f8a-a9c4-458c-ba81-e61970f18cdf-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:12:35.305955 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.305886    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwtm8\" (UniqueName: \"kubernetes.io/projected/0aae3bb8-04d9-40e1-bcea-476334fa6363-kube-api-access-kwtm8\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:12:35.307766 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.307733    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/100a0f8a-a9c4-458c-ba81-e61970f18cdf-kube-api-access-75zcw" pod "100a0f8a-a9c4-458c-ba81-e61970f18cdf" (UID: "100a0f8a-a9c4-458c-ba81-e61970f18cdf"). InnerVolumeSpecName "kube-api-access-75zcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:12:35.407417 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.407380    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75zcw\" (UniqueName: \"kubernetes.io/projected/100a0f8a-a9c4-458c-ba81-e61970f18cdf-kube-api-access-75zcw\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:12:35.428091 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.428052    4327 generic.go:356] "Generic (PLEG): container finished" podID="100a0f8a-a9c4-458c-ba81-e61970f18cdf" containerID="f5bdd7017942890bbdb7dd5e3b774a18c10c3456e70077a2e3f543880f7d3186" exitCode=0
Oct 31 01:12:35.428271 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.428106    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54"
Oct 31 01:12:35.428271 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.428145    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54" event={"ID":"100a0f8a-a9c4-458c-ba81-e61970f18cdf","Type":"ContainerDied","Data":"f5bdd7017942890bbdb7dd5e3b774a18c10c3456e70077a2e3f543880f7d3186"}
Oct 31 01:12:35.428271 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.428222    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54" event={"ID":"100a0f8a-a9c4-458c-ba81-e61970f18cdf","Type":"ContainerDied","Data":"fda7958e90eeac57cd599a75b0385cac3f74d4b7d6e0e93febf3e2dfa847ebc5"}
Oct 31 01:12:35.428271 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.428244    4327 scope.go:122] "RemoveContainer" containerID="f5bdd7017942890bbdb7dd5e3b774a18c10c3456e70077a2e3f543880f7d3186"
Oct 31 01:12:35.430019 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.429992    4327 generic.go:356] "Generic (PLEG): container finished" podID="0aae3bb8-04d9-40e1-bcea-476334fa6363" containerID="86679373ac565aec247c997de393c568f0e02d4e6ddb961cdfb313868342289f" exitCode=0
Oct 31 01:12:35.430134 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.430046    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm"
Oct 31 01:12:35.430134 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.430049    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm" event={"ID":"0aae3bb8-04d9-40e1-bcea-476334fa6363","Type":"ContainerDied","Data":"86679373ac565aec247c997de393c568f0e02d4e6ddb961cdfb313868342289f"}
Oct 31 01:12:35.430223 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.430145    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm" event={"ID":"0aae3bb8-04d9-40e1-bcea-476334fa6363","Type":"ContainerDied","Data":"6a9a805e589145bdb7a40ce66fd32d6d290df086da8ef47b3836c8c330691c70"}
Oct 31 01:12:35.441076 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.441057    4327 scope.go:122] "RemoveContainer" containerID="f5bdd7017942890bbdb7dd5e3b774a18c10c3456e70077a2e3f543880f7d3186"
Oct 31 01:12:35.441349 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:12:35.441331    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5bdd7017942890bbdb7dd5e3b774a18c10c3456e70077a2e3f543880f7d3186\": container with ID starting with f5bdd7017942890bbdb7dd5e3b774a18c10c3456e70077a2e3f543880f7d3186 not found: ID does not exist" containerID="f5bdd7017942890bbdb7dd5e3b774a18c10c3456e70077a2e3f543880f7d3186"
Oct 31 01:12:35.441398 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.441356    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5bdd7017942890bbdb7dd5e3b774a18c10c3456e70077a2e3f543880f7d3186"} err="failed to get container status \"f5bdd7017942890bbdb7dd5e3b774a18c10c3456e70077a2e3f543880f7d3186\": rpc error: code = NotFound desc = could not find container \"f5bdd7017942890bbdb7dd5e3b774a18c10c3456e70077a2e3f543880f7d3186\": container with ID starting with f5bdd7017942890bbdb7dd5e3b774a18c10c3456e70077a2e3f543880f7d3186 not found: ID does not exist"
Oct 31 01:12:35.441398 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.441373    4327 scope.go:122] "RemoveContainer" containerID="86679373ac565aec247c997de393c568f0e02d4e6ddb961cdfb313868342289f"
Oct 31 01:12:35.450642 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.450625    4327 scope.go:122] "RemoveContainer" containerID="86679373ac565aec247c997de393c568f0e02d4e6ddb961cdfb313868342289f"
Oct 31 01:12:35.450933 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:12:35.450915    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86679373ac565aec247c997de393c568f0e02d4e6ddb961cdfb313868342289f\": container with ID starting with 86679373ac565aec247c997de393c568f0e02d4e6ddb961cdfb313868342289f not found: ID does not exist" containerID="86679373ac565aec247c997de393c568f0e02d4e6ddb961cdfb313868342289f"
Oct 31 01:12:35.451011 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.450939    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86679373ac565aec247c997de393c568f0e02d4e6ddb961cdfb313868342289f"} err="failed to get container status \"86679373ac565aec247c997de393c568f0e02d4e6ddb961cdfb313868342289f\": rpc error: code = NotFound desc = could not find container \"86679373ac565aec247c997de393c568f0e02d4e6ddb961cdfb313868342289f\": container with ID starting with 86679373ac565aec247c997de393c568f0e02d4e6ddb961cdfb313868342289f not found: ID does not exist"
Oct 31 01:12:35.480666 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.480627    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54"]
Oct 31 01:12:35.492809 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.492780    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-f9ccf6c5-qtr54"]
Oct 31 01:12:35.526390 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.526349    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm"]
Oct 31 01:12:35.531018 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.530986    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75ead8-5b87c69f95-vt2vm"]
Oct 31 01:12:35.923210 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.923180    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aae3bb8-04d9-40e1-bcea-476334fa6363" path="/var/lib/kubelet/pods/0aae3bb8-04d9-40e1-bcea-476334fa6363/volumes"
Oct 31 01:12:35.923612 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:35.923598    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="100a0f8a-a9c4-458c-ba81-e61970f18cdf" path="/var/lib/kubelet/pods/100a0f8a-a9c4-458c-ba81-e61970f18cdf/volumes"
Oct 31 01:12:40.388614 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:40.388576    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z"
Oct 31 01:12:42.404898 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:12:42.404858    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d"
Oct 31 01:12:45.293837 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:12:45.293795    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:13:03.343968 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:03.343929    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc"]
Oct 31 01:13:03.351044 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:03.351001    4327 cpu_manager.go:414] "containerMap: removing container" podUID="100a0f8a-a9c4-458c-ba81-e61970f18cdf" containerName="payload-pre-processing"
Oct 31 01:13:03.351044 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:03.351038    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="100a0f8a-a9c4-458c-ba81-e61970f18cdf" containerName="payload-pre-processing"
Oct 31 01:13:03.351290 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:03.351068    4327 cpu_manager.go:414] "containerMap: removing container" podUID="0aae3bb8-04d9-40e1-bcea-476334fa6363" containerName="payload-processing"
Oct 31 01:13:03.351290 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:03.351076    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aae3bb8-04d9-40e1-bcea-476334fa6363" containerName="payload-processing"
Oct 31 01:13:03.351388 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:03.351344    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="100a0f8a-a9c4-458c-ba81-e61970f18cdf" containerName="payload-pre-processing"
Oct 31 01:13:03.351388 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:03.351368    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aae3bb8-04d9-40e1-bcea-476334fa6363" containerName="payload-processing"
Oct 31 01:13:03.355484 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:03.355461    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc"
Oct 31 01:13:03.365585 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:03.365555    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc"]
Oct 31 01:13:03.378707 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:03.376861    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-75ead8\""
Oct 31 01:13:03.389380 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:03.389346    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-75ead8-dockercfg-7bngp\""
Oct 31 01:13:03.407546 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:03.397276    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98668443-03d5-49cf-9c91-29afc4bf3447-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc\" (UID: \"98668443-03d5-49cf-9c91-29afc4bf3447\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc"
Oct 31 01:13:03.407546 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:03.397389    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brn78\" (UniqueName: \"kubernetes.io/projected/98668443-03d5-49cf-9c91-29afc4bf3447-kube-api-access-brn78\") pod \"payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc\" (UID: \"98668443-03d5-49cf-9c91-29afc4bf3447\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc"
Oct 31 01:13:03.498762 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:03.498730    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brn78\" (UniqueName: \"kubernetes.io/projected/98668443-03d5-49cf-9c91-29afc4bf3447-kube-api-access-brn78\") pod \"payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc\" (UID: \"98668443-03d5-49cf-9c91-29afc4bf3447\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc"
Oct 31 01:13:03.498961 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:03.498945    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98668443-03d5-49cf-9c91-29afc4bf3447-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc\" (UID: \"98668443-03d5-49cf-9c91-29afc4bf3447\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc"
Oct 31 01:13:03.499647 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:03.499628    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98668443-03d5-49cf-9c91-29afc4bf3447-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc\" (UID: \"98668443-03d5-49cf-9c91-29afc4bf3447\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc"
Oct 31 01:13:03.523803 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:03.523770    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brn78\" (UniqueName: \"kubernetes.io/projected/98668443-03d5-49cf-9c91-29afc4bf3447-kube-api-access-brn78\") pod \"payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc\" (UID: \"98668443-03d5-49cf-9c91-29afc4bf3447\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc"
Oct 31 01:13:03.693469 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:03.693439    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc"
Oct 31 01:13:03.953038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:03.952958    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t"]
Oct 31 01:13:03.958493 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:03.958469    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t"
Oct 31 01:13:04.006764 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:04.005168    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-e41918-serving-cert\""
Oct 31 01:13:04.010113 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:04.007028    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t\" (UID: \"fc74ef25-794f-4410-b3b7-403c7ba9f7d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t"
Oct 31 01:13:04.010113 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:04.007174    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t\" (UID: \"fc74ef25-794f-4410-b3b7-403c7ba9f7d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t"
Oct 31 01:13:04.010113 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:04.007366    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c4kz\" (UniqueName: \"kubernetes.io/projected/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-kube-api-access-5c4kz\") pod \"maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t\" (UID: \"fc74ef25-794f-4410-b3b7-403c7ba9f7d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t"
Oct 31 01:13:04.010113 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:04.007423    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-metrics-certs\") pod \"maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t\" (UID: \"fc74ef25-794f-4410-b3b7-403c7ba9f7d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t"
Oct 31 01:13:04.013786 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:04.013755    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t"]
Oct 31 01:13:04.049945 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:04.049914    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc"]
Oct 31 01:13:04.065727 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:13:04.065689    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98668443_03d5_49cf_9c91_29afc4bf3447.slice/crio-8afe741762c8438fd4971d557b1521c07751f0bc6e8c9ac88ac5e643e88bf37f WatchSource:0}: Error finding container 8afe741762c8438fd4971d557b1521c07751f0bc6e8c9ac88ac5e643e88bf37f: Status 404 returned error can't find the container with id 8afe741762c8438fd4971d557b1521c07751f0bc6e8c9ac88ac5e643e88bf37f
Oct 31 01:13:04.071211 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:04.071191    4327 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:13:04.108822 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:04.108785    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c4kz\" (UniqueName: \"kubernetes.io/projected/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-kube-api-access-5c4kz\") pod \"maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t\" (UID: \"fc74ef25-794f-4410-b3b7-403c7ba9f7d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t"
Oct 31 01:13:04.109050 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:04.108841    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-metrics-certs\") pod \"maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t\" (UID: \"fc74ef25-794f-4410-b3b7-403c7ba9f7d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t"
Oct 31 01:13:04.109050 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:04.108995    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t\" (UID: \"fc74ef25-794f-4410-b3b7-403c7ba9f7d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t"
Oct 31 01:13:04.109182 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:04.109119    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t\" (UID: \"fc74ef25-794f-4410-b3b7-403c7ba9f7d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t"
Oct 31 01:13:04.109957 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:04.109933    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t\" (UID: \"fc74ef25-794f-4410-b3b7-403c7ba9f7d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t"
Oct 31 01:13:04.112033 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:04.112006    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t\" (UID: \"fc74ef25-794f-4410-b3b7-403c7ba9f7d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t"
Oct 31 01:13:04.112131 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:04.112058    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-metrics-certs\") pod \"maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t\" (UID: \"fc74ef25-794f-4410-b3b7-403c7ba9f7d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t"
Oct 31 01:13:04.136945 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:04.136904    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c4kz\" (UniqueName: \"kubernetes.io/projected/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-kube-api-access-5c4kz\") pod \"maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t\" (UID: \"fc74ef25-794f-4410-b3b7-403c7ba9f7d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t"
Oct 31 01:13:04.277809 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:04.277665    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t"
Oct 31 01:13:04.599718 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:04.593193    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t"]
Oct 31 01:13:04.717759 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:04.717716    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc" event={"ID":"98668443-03d5-49cf-9c91-29afc4bf3447","Type":"ContainerStarted","Data":"8afe741762c8438fd4971d557b1521c07751f0bc6e8c9ac88ac5e643e88bf37f"}
Oct 31 01:13:04.719525 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:04.719493    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t" event={"ID":"fc74ef25-794f-4410-b3b7-403c7ba9f7d8","Type":"ContainerStarted","Data":"31e0e42208af52853fbd2e4888298f39874ee8fcc3c7582421f7fe2fea0f6a5e"}
Oct 31 01:13:05.732016 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:05.731976    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t" event={"ID":"fc74ef25-794f-4410-b3b7-403c7ba9f7d8","Type":"ContainerStarted","Data":"e3d33b14d748db76712767ff0171bbd9aeb76962dd305f51ccb35a9b074468aa"}
Oct 31 01:13:05.732472 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:05.732052    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t"
Oct 31 01:13:05.776287 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:05.776233    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t" podStartSLOduration=2.375521112 podStartE2EDuration="2.776218534s" totalImagesPullingTime="400.697422ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:05.774827891 +0000 UTC m=+3245.996863434" watchObservedRunningTime="2026-10-31 01:13:05.776218534 +0000 UTC m=+3245.998254076"
Oct 31 01:13:06.746506 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:06.746456    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc" event={"ID":"98668443-03d5-49cf-9c91-29afc4bf3447","Type":"ContainerStarted","Data":"2b89019eae061948c94a5e2d6ed625c62cf3be8c807bcf64743a5c2199f25f2f"}
Oct 31 01:13:06.746989 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:06.746569    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc"
Oct 31 01:13:06.790407 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:06.790351    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc" podStartSLOduration=2.067571553 podStartE2EDuration="3.79033571s" totalImagesPullingTime="1.722764157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:06.788139685 +0000 UTC m=+3247.010175229" watchObservedRunningTime="2026-10-31 01:13:06.79033571 +0000 UTC m=+3247.012371295"
Oct 31 01:13:11.753256 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:11.753226    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t"
Oct 31 01:13:15.294498 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:13:15.294446    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:13:17.756657 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:17.756613    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc"
Oct 31 01:13:20.181761 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:20.181722    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc"]
Oct 31 01:13:20.186639 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:20.186613    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc"
Oct 31 01:13:20.215670 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:20.215632    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc"]
Oct 31 01:13:20.226077 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:20.226041    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4207cd-dockercfg-q4hc9\""
Oct 31 01:13:20.226277 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:20.226149    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4207cd\""
Oct 31 01:13:20.263564 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:20.263521    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq5mb\" (UniqueName: \"kubernetes.io/projected/ede414eb-a19a-4753-9317-dcbbdca0ecf1-kube-api-access-jq5mb\") pod \"payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc\" (UID: \"ede414eb-a19a-4753-9317-dcbbdca0ecf1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc"
Oct 31 01:13:20.264363 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:20.264137    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ede414eb-a19a-4753-9317-dcbbdca0ecf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc\" (UID: \"ede414eb-a19a-4753-9317-dcbbdca0ecf1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc"
Oct 31 01:13:20.305988 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:20.305832    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w"]
Oct 31 01:13:20.314081 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:20.314053    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w"
Oct 31 01:13:20.338282 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:20.338247    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w"]
Oct 31 01:13:20.365912 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:20.365873    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ede414eb-a19a-4753-9317-dcbbdca0ecf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc\" (UID: \"ede414eb-a19a-4753-9317-dcbbdca0ecf1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc"
Oct 31 01:13:20.366113 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:20.366036    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq5mb\" (UniqueName: \"kubernetes.io/projected/ede414eb-a19a-4753-9317-dcbbdca0ecf1-kube-api-access-jq5mb\") pod \"payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc\" (UID: \"ede414eb-a19a-4753-9317-dcbbdca0ecf1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc"
Oct 31 01:13:20.367383 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:20.367354    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ede414eb-a19a-4753-9317-dcbbdca0ecf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc\" (UID: \"ede414eb-a19a-4753-9317-dcbbdca0ecf1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc"
Oct 31 01:13:20.429704 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:20.421315    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq5mb\" (UniqueName: \"kubernetes.io/projected/ede414eb-a19a-4753-9317-dcbbdca0ecf1-kube-api-access-jq5mb\") pod \"payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc\" (UID: \"ede414eb-a19a-4753-9317-dcbbdca0ecf1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc"
Oct 31 01:13:20.468335 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:20.468230    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96158b4f-c939-403a-9e2f-c037ebc5cf0e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w\" (UID: \"96158b4f-c939-403a-9e2f-c037ebc5cf0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w"
Oct 31 01:13:20.468526 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:20.468380    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22tpl\" (UniqueName: \"kubernetes.io/projected/96158b4f-c939-403a-9e2f-c037ebc5cf0e-kube-api-access-22tpl\") pod \"payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w\" (UID: \"96158b4f-c939-403a-9e2f-c037ebc5cf0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w"
Oct 31 01:13:20.508256 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:20.508224    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc"
Oct 31 01:13:20.569635 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:20.569597    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22tpl\" (UniqueName: \"kubernetes.io/projected/96158b4f-c939-403a-9e2f-c037ebc5cf0e-kube-api-access-22tpl\") pod \"payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w\" (UID: \"96158b4f-c939-403a-9e2f-c037ebc5cf0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w"
Oct 31 01:13:20.569904 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:20.569885    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96158b4f-c939-403a-9e2f-c037ebc5cf0e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w\" (UID: \"96158b4f-c939-403a-9e2f-c037ebc5cf0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w"
Oct 31 01:13:20.570628 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:20.570603    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96158b4f-c939-403a-9e2f-c037ebc5cf0e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w\" (UID: \"96158b4f-c939-403a-9e2f-c037ebc5cf0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w"
Oct 31 01:13:20.595326 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:20.595262    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22tpl\" (UniqueName: \"kubernetes.io/projected/96158b4f-c939-403a-9e2f-c037ebc5cf0e-kube-api-access-22tpl\") pod \"payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w\" (UID: \"96158b4f-c939-403a-9e2f-c037ebc5cf0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w"
Oct 31 01:13:20.664022 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:20.663986    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w"
Oct 31 01:13:20.813509 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:13:20.813475    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede414eb_a19a_4753_9317_dcbbdca0ecf1.slice/crio-151b8a4bc26a520c38902adbb77874e67b7486f918425a15dc2400f0b2fbfdda WatchSource:0}: Error finding container 151b8a4bc26a520c38902adbb77874e67b7486f918425a15dc2400f0b2fbfdda: Status 404 returned error can't find the container with id 151b8a4bc26a520c38902adbb77874e67b7486f918425a15dc2400f0b2fbfdda
Oct 31 01:13:20.814400 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:20.814377    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc"]
Oct 31 01:13:20.864224 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:20.864192    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc" event={"ID":"ede414eb-a19a-4753-9317-dcbbdca0ecf1","Type":"ContainerStarted","Data":"151b8a4bc26a520c38902adbb77874e67b7486f918425a15dc2400f0b2fbfdda"}
Oct 31 01:13:20.964286 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:13:20.964251    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96158b4f_c939_403a_9e2f_c037ebc5cf0e.slice/crio-d7d926cd73b6fa2d52c45e471dc28a9fea32cca8ade4c382435a89947edd76c4 WatchSource:0}: Error finding container d7d926cd73b6fa2d52c45e471dc28a9fea32cca8ade4c382435a89947edd76c4: Status 404 returned error can't find the container with id d7d926cd73b6fa2d52c45e471dc28a9fea32cca8ade4c382435a89947edd76c4
Oct 31 01:13:20.965688 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:20.965655    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w"]
Oct 31 01:13:21.874433 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:21.874387    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc" event={"ID":"ede414eb-a19a-4753-9317-dcbbdca0ecf1","Type":"ContainerStarted","Data":"b394cee5815c523086c53b43993a7c66a7f6afa8f00d1c4c140c18422ca0d2dc"}
Oct 31 01:13:21.874949 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:21.874464    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc"
Oct 31 01:13:21.877196 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:21.877162    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w" event={"ID":"96158b4f-c939-403a-9e2f-c037ebc5cf0e","Type":"ContainerStarted","Data":"cf063eb96950b0bc3b361f4c589df04e4b5549f6c45693c751c9dd5cce14c418"}
Oct 31 01:13:21.877332 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:21.877220    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w" event={"ID":"96158b4f-c939-403a-9e2f-c037ebc5cf0e","Type":"ContainerStarted","Data":"d7d926cd73b6fa2d52c45e471dc28a9fea32cca8ade4c382435a89947edd76c4"}
Oct 31 01:13:21.877332 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:21.877249    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w"
Oct 31 01:13:21.918013 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:21.917966    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc" podStartSLOduration=1.917952461 podStartE2EDuration="1.917952461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:21.914910037 +0000 UTC m=+3262.136945580" watchObservedRunningTime="2026-10-31 01:13:21.917952461 +0000 UTC m=+3262.139988003"
Oct 31 01:13:21.962882 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:21.962830    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w" podStartSLOduration=1.962815859 podStartE2EDuration="1.962815859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:21.95926707 +0000 UTC m=+3262.181302614" watchObservedRunningTime="2026-10-31 01:13:21.962815859 +0000 UTC m=+3262.184851400"
Oct 31 01:13:32.887873 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:32.887835    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc"
Oct 31 01:13:32.888309 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:32.887944    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w"
Oct 31 01:13:38.611570 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:38.611534    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"]
Oct 31 01:13:38.612124 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:38.611756    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx" podUID="d93e1303-7225-4f18-b344-ffeea15da3b5" containerName="istio-proxy" containerID="cri-o://e540b830445d04d603f1c5ae3c06717d29902873a0c4413b8ddf7736ce5815fb" gracePeriod=30
Oct 31 01:13:41.517859 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.517820    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"]
Oct 31 01:13:41.524921 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.524895    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.543629 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.543605    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-bcdb4e-openshift-default-dockercfg-s86d8\""
Oct 31 01:13:41.560621 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.560585    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"]
Oct 31 01:13:41.588886 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.588816    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6879302-3938-426c-a79b-8738dee89964-istiod-ca-cert\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.589099 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.588985    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-istio-data\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.589099 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.589059    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6879302-3938-426c-a79b-8738dee89964-istio-token\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.589230 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.589208    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-workload-socket\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.589328 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.589310    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6879302-3938-426c-a79b-8738dee89964-istio-podinfo\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.589470 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.589443    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhqtn\" (UniqueName: \"kubernetes.io/projected/c6879302-3938-426c-a79b-8738dee89964-kube-api-access-dhqtn\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.589610 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.589584    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-workload-certs\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.589879 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.589845    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-credential-socket\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.589970 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.589891    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-istio-envoy\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.690796 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.690757    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-workload-socket\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.691003 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.690845    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6879302-3938-426c-a79b-8738dee89964-istio-podinfo\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.691003 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.690898    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhqtn\" (UniqueName: \"kubernetes.io/projected/c6879302-3938-426c-a79b-8738dee89964-kube-api-access-dhqtn\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.691003 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.690958    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-workload-certs\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.691003 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.690997    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-credential-socket\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.691248 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.691020    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-istio-envoy\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.691248 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.691051    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6879302-3938-426c-a79b-8738dee89964-istiod-ca-cert\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.691248 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.691085    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-istio-data\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.691248 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.691138    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6879302-3938-426c-a79b-8738dee89964-istio-token\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.691421 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.691261    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-workload-socket\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.691421 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.691364    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-workload-certs\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.691507 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.691419    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-credential-socket\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.691658 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.691635    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-istio-data\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.691888 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.691862    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6879302-3938-426c-a79b-8738dee89964-istiod-ca-cert\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.693405 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.693383    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-istio-envoy\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.693505 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.693493    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6879302-3938-426c-a79b-8738dee89964-istio-podinfo\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.714560 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.714530    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6879302-3938-426c-a79b-8738dee89964-istio-token\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.714730 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.714663    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhqtn\" (UniqueName: \"kubernetes.io/projected/c6879302-3938-426c-a79b-8738dee89964-kube-api-access-dhqtn\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:41.841705 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:41.841594    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:42.165307 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:13:42.165273    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6879302_3938_426c_a79b_8738dee89964.slice/crio-5b17498c45b277f7a508b5e611aa4d20f1f65ca626d676d7918789a0863d7618 WatchSource:0}: Error finding container 5b17498c45b277f7a508b5e611aa4d20f1f65ca626d676d7918789a0863d7618: Status 404 returned error can't find the container with id 5b17498c45b277f7a508b5e611aa4d20f1f65ca626d676d7918789a0863d7618
Oct 31 01:13:42.165485 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:42.165468    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"]
Oct 31 01:13:42.168313 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:42.168280    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:13:42.168405 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:42.168350    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:13:42.168405 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:42.168381    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:13:43.035985 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:43.035945    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s" event={"ID":"c6879302-3938-426c-a79b-8738dee89964","Type":"ContainerStarted","Data":"9a463c5dfb650be76fdde49ef85419e7cda82b7de583521b94d28ecd2e4a57f4"}
Oct 31 01:13:43.036411 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:43.036014    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s" event={"ID":"c6879302-3938-426c-a79b-8738dee89964","Type":"ContainerStarted","Data":"5b17498c45b277f7a508b5e611aa4d20f1f65ca626d676d7918789a0863d7618"}
Oct 31 01:13:43.101057 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:43.100991    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s" podStartSLOduration=2.100972215 podStartE2EDuration="2.100972215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:43.098090794 +0000 UTC m=+3283.320126365" watchObservedRunningTime="2026-10-31 01:13:43.100972215 +0000 UTC m=+3283.323007757"
Oct 31 01:13:43.852384 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:43.852149    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 31 01:13:43.852384 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:43.852225    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s" podUID="c6879302-3938-426c-a79b-8738dee89964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 31 01:13:43.853863 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:43.852701    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:43.952589 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:43.952557    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 01:13:44.057034 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.056996    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-credential-socket\") pod \"d93e1303-7225-4f18-b344-ffeea15da3b5\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") "
Oct 31 01:13:44.057434 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.057146    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-data\") pod \"d93e1303-7225-4f18-b344-ffeea15da3b5\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") "
Oct 31 01:13:44.057434 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.057190    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d93e1303-7225-4f18-b344-ffeea15da3b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d93e1303-7225-4f18-b344-ffeea15da3b5-istiod-ca-cert\") pod \"d93e1303-7225-4f18-b344-ffeea15da3b5\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") "
Oct 31 01:13:44.057434 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.057270    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-token\" (UniqueName: \"kubernetes.io/projected/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-token\") pod \"d93e1303-7225-4f18-b344-ffeea15da3b5\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") "
Oct 31 01:13:44.057434 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.057305    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-podinfo\") pod \"d93e1303-7225-4f18-b344-ffeea15da3b5\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") "
Oct 31 01:13:44.057434 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.057362    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-workload-socket\") pod \"d93e1303-7225-4f18-b344-ffeea15da3b5\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") "
Oct 31 01:13:44.057608 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.057499    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d93e1303-7225-4f18-b344-ffeea15da3b5-kube-api-access-pkpkz\" (UniqueName: \"kubernetes.io/projected/d93e1303-7225-4f18-b344-ffeea15da3b5-kube-api-access-pkpkz\") pod \"d93e1303-7225-4f18-b344-ffeea15da3b5\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") "
Oct 31 01:13:44.057608 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.057564    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-envoy\") pod \"d93e1303-7225-4f18-b344-ffeea15da3b5\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") "
Oct 31 01:13:44.057693 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.057622    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-workload-certs\") pod \"d93e1303-7225-4f18-b344-ffeea15da3b5\" (UID: \"d93e1303-7225-4f18-b344-ffeea15da3b5\") "
Oct 31 01:13:44.060834 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.060800    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-workload-certs" pod "d93e1303-7225-4f18-b344-ffeea15da3b5" (UID: "d93e1303-7225-4f18-b344-ffeea15da3b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:13:44.061168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.061145    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-credential-socket" pod "d93e1303-7225-4f18-b344-ffeea15da3b5" (UID: "d93e1303-7225-4f18-b344-ffeea15da3b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:13:44.061362 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.061339    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-data" pod "d93e1303-7225-4f18-b344-ffeea15da3b5" (UID: "d93e1303-7225-4f18-b344-ffeea15da3b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:13:44.061665 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.061642    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d93e1303-7225-4f18-b344-ffeea15da3b5-istiod-ca-cert" pod "d93e1303-7225-4f18-b344-ffeea15da3b5" (UID: "d93e1303-7225-4f18-b344-ffeea15da3b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:13:44.069877 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.069570    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-workload-socket" pod "d93e1303-7225-4f18-b344-ffeea15da3b5" (UID: "d93e1303-7225-4f18-b344-ffeea15da3b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:13:44.075527 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.075368    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-token" pod "d93e1303-7225-4f18-b344-ffeea15da3b5" (UID: "d93e1303-7225-4f18-b344-ffeea15da3b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:13:44.075527 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.075472    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-envoy" pod "d93e1303-7225-4f18-b344-ffeea15da3b5" (UID: "d93e1303-7225-4f18-b344-ffeea15da3b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:13:44.076293 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.076264    4327 generic.go:356] "Generic (PLEG): container finished" podID="d93e1303-7225-4f18-b344-ffeea15da3b5" containerID="e540b830445d04d603f1c5ae3c06717d29902873a0c4413b8ddf7736ce5815fb" exitCode=0
Oct 31 01:13:44.076387 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.076369    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"
Oct 31 01:13:44.076550 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.076525    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx" event={"ID":"d93e1303-7225-4f18-b344-ffeea15da3b5","Type":"ContainerDied","Data":"e540b830445d04d603f1c5ae3c06717d29902873a0c4413b8ddf7736ce5815fb"}
Oct 31 01:13:44.076604 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.076581    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx" event={"ID":"d93e1303-7225-4f18-b344-ffeea15da3b5","Type":"ContainerDied","Data":"3e3b107947d93b88caccd421e947044889b0bdf05c2734ed941b8a16e9b95c08"}
Oct 31 01:13:44.076659 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.076611    4327 scope.go:122] "RemoveContainer" containerID="e540b830445d04d603f1c5ae3c06717d29902873a0c4413b8ddf7736ce5815fb"
Oct 31 01:13:44.078620 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.078589    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-podinfo" pod "d93e1303-7225-4f18-b344-ffeea15da3b5" (UID: "d93e1303-7225-4f18-b344-ffeea15da3b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:13:44.079927 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.079905    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d93e1303-7225-4f18-b344-ffeea15da3b5-kube-api-access-pkpkz" pod "d93e1303-7225-4f18-b344-ffeea15da3b5" (UID: "d93e1303-7225-4f18-b344-ffeea15da3b5"). InnerVolumeSpecName "kube-api-access-pkpkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:13:44.139705 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.116778    4327 scope.go:122] "RemoveContainer" containerID="e39dc35f9909ff40d3b9b5da429e6637263a9c3ff9f0c156b6628c0d33a63bfd"
Oct 31 01:13:44.159765 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.159535    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:13:44.159765 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.159564    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:13:44.159765 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.159576    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkpkz\" (UniqueName: \"kubernetes.io/projected/d93e1303-7225-4f18-b344-ffeea15da3b5-kube-api-access-pkpkz\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:13:44.159765 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.159601    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:13:44.159765 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.159611    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:13:44.159765 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.159622    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:13:44.159765 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.159634    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:13:44.159765 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.159646    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d93e1303-7225-4f18-b344-ffeea15da3b5-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:13:44.159765 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.159658    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d93e1303-7225-4f18-b344-ffeea15da3b5-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:13:44.164910 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.164882    4327 scope.go:122] "RemoveContainer" containerID="e540b830445d04d603f1c5ae3c06717d29902873a0c4413b8ddf7736ce5815fb"
Oct 31 01:13:44.165293 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:13:44.165266    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e540b830445d04d603f1c5ae3c06717d29902873a0c4413b8ddf7736ce5815fb\": container with ID starting with e540b830445d04d603f1c5ae3c06717d29902873a0c4413b8ddf7736ce5815fb not found: ID does not exist" containerID="e540b830445d04d603f1c5ae3c06717d29902873a0c4413b8ddf7736ce5815fb"
Oct 31 01:13:44.165373 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.165302    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e540b830445d04d603f1c5ae3c06717d29902873a0c4413b8ddf7736ce5815fb"} err="failed to get container status \"e540b830445d04d603f1c5ae3c06717d29902873a0c4413b8ddf7736ce5815fb\": rpc error: code = NotFound desc = could not find container \"e540b830445d04d603f1c5ae3c06717d29902873a0c4413b8ddf7736ce5815fb\": container with ID starting with e540b830445d04d603f1c5ae3c06717d29902873a0c4413b8ddf7736ce5815fb not found: ID does not exist"
Oct 31 01:13:44.165373 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.165321    4327 scope.go:122] "RemoveContainer" containerID="e39dc35f9909ff40d3b9b5da429e6637263a9c3ff9f0c156b6628c0d33a63bfd"
Oct 31 01:13:44.165607 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:13:44.165587    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e39dc35f9909ff40d3b9b5da429e6637263a9c3ff9f0c156b6628c0d33a63bfd\": container with ID starting with e39dc35f9909ff40d3b9b5da429e6637263a9c3ff9f0c156b6628c0d33a63bfd not found: ID does not exist" containerID="e39dc35f9909ff40d3b9b5da429e6637263a9c3ff9f0c156b6628c0d33a63bfd"
Oct 31 01:13:44.165666 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.165615    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39dc35f9909ff40d3b9b5da429e6637263a9c3ff9f0c156b6628c0d33a63bfd"} err="failed to get container status \"e39dc35f9909ff40d3b9b5da429e6637263a9c3ff9f0c156b6628c0d33a63bfd\": rpc error: code = NotFound desc = could not find container \"e39dc35f9909ff40d3b9b5da429e6637263a9c3ff9f0c156b6628c0d33a63bfd\": container with ID starting with e39dc35f9909ff40d3b9b5da429e6637263a9c3ff9f0c156b6628c0d33a63bfd not found: ID does not exist"
Oct 31 01:13:44.442705 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.436751    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"]
Oct 31 01:13:44.448840 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.448809    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pq9nx"]
Oct 31 01:13:44.842425 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.842201    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 31 01:13:44.842425 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:44.842278    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s" podUID="c6879302-3938-426c-a79b-8738dee89964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 31 01:13:45.296994 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:13:45.296955    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:13:45.842186 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:45.842138    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 31 01:13:45.842418 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:45.842203    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s" podUID="c6879302-3938-426c-a79b-8738dee89964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 31 01:13:45.947477 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:45.947440    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d93e1303-7225-4f18-b344-ffeea15da3b5" path="/var/lib/kubelet/pods/d93e1303-7225-4f18-b344-ffeea15da3b5/volumes"
Oct 31 01:13:46.842204 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:46.842151    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 31 01:13:46.842618 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:46.842247    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s" podUID="c6879302-3938-426c-a79b-8738dee89964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 31 01:13:47.842118 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:47.842077    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 31 01:13:47.842336 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:47.842150    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s" podUID="c6879302-3938-426c-a79b-8738dee89964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 31 01:13:48.842172 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:48.842130    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 31 01:13:48.842388 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:48.842203    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s" podUID="c6879302-3938-426c-a79b-8738dee89964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 31 01:13:49.270876 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:49.270832    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d"]
Oct 31 01:13:49.307429 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:49.307391    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z"]
Oct 31 01:13:49.848008 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:49.847970    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:49.848490 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:49.848447    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:49.849105 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:49.849086    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:13:50.442264 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.442227    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"]
Oct 31 01:13:50.443462 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.443432    4327 cpu_manager.go:414] "containerMap: removing container" podUID="d93e1303-7225-4f18-b344-ffeea15da3b5" containerName="istio-proxy"
Oct 31 01:13:50.443462 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.443462    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d93e1303-7225-4f18-b344-ffeea15da3b5" containerName="istio-proxy"
Oct 31 01:13:50.443661 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.443499    4327 cpu_manager.go:414] "containerMap: removing container" podUID="d93e1303-7225-4f18-b344-ffeea15da3b5" containerName="istio-proxy"
Oct 31 01:13:50.443661 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.443507    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d93e1303-7225-4f18-b344-ffeea15da3b5" containerName="istio-proxy"
Oct 31 01:13:50.443798 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.443672    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="d93e1303-7225-4f18-b344-ffeea15da3b5" containerName="istio-proxy"
Oct 31 01:13:50.449398 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.449366    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.476973 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.476929    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"]
Oct 31 01:13:50.483617 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.483568    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-33d44f-openshift-default-dockercfg-qnvnz\""
Oct 31 01:13:50.500737 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.500657    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-credential-socket\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.500928 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.500766    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-workload-socket\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.500928 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.500829    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-data\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.500928 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.500866    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-token\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.500928 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.500917    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-workload-certs\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.501151 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.501018    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-podinfo\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.501151 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.501091    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istiod-ca-cert\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.501151 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.501139    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-envoy\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.501313 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.501190    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdxmj\" (UniqueName: \"kubernetes.io/projected/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-kube-api-access-kdxmj\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.602625 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.602580    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-credential-socket\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.602868 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.602690    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-workload-socket\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.602868 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.602745    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-data\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.602868 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.602775    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-token\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.602868 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.602821    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-workload-certs\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.603122 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.602926    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-podinfo\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.603122 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.602998    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istiod-ca-cert\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.603122 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.603042    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-envoy\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.603122 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.603086    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdxmj\" (UniqueName: \"kubernetes.io/projected/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-kube-api-access-kdxmj\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.603364 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.603332    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-data\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.603549 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.603530    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-credential-socket\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.603775 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.603661    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-workload-socket\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.603775 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.603763    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-workload-certs\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.604216 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.604195    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istiod-ca-cert\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.605673 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.605653    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-envoy\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.606186 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.606166    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-podinfo\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.629446 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.629419    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-token\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.629666 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.629520    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdxmj\" (UniqueName: \"kubernetes.io/projected/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-kube-api-access-kdxmj\") pod \"e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") " pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:50.792769 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:50.781928    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:51.152610 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:13:51.152583    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae467a1_6a9a_4c1b_855a_ac8edb1d10ee.slice/crio-bbd8b494201e60b1c944c23a02a5f7bb1756353110fd0b80b7511797d223d9a3 WatchSource:0}: Error finding container bbd8b494201e60b1c944c23a02a5f7bb1756353110fd0b80b7511797d223d9a3: Status 404 returned error can't find the container with id bbd8b494201e60b1c944c23a02a5f7bb1756353110fd0b80b7511797d223d9a3
Oct 31 01:13:51.153514 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:51.153488    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"]
Oct 31 01:13:51.155352 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:51.155313    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:13:51.155465 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:51.155393    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:13:51.155465 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:51.155438    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:13:52.141897 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:52.141857    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2" event={"ID":"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee","Type":"ContainerStarted","Data":"30ab02190365330996820b709bd42274b2e961ee11f70d487aa8a90bcb827acd"}
Oct 31 01:13:52.142093 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:52.141910    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2" event={"ID":"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee","Type":"ContainerStarted","Data":"bbd8b494201e60b1c944c23a02a5f7bb1756353110fd0b80b7511797d223d9a3"}
Oct 31 01:13:52.204352 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:52.204295    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2" podStartSLOduration=2.204280079 podStartE2EDuration="2.204280079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:52.202370171 +0000 UTC m=+3292.424405713" watchObservedRunningTime="2026-10-31 01:13:52.204280079 +0000 UTC m=+3292.426315620"
Oct 31 01:13:52.782868 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:52.782837    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:52.789172 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:52.789145    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:53.149426 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:53.149391    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:53.150429 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:53.150410    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:13:54.271195 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:54.271135    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d" podUID="6d7550dd-2f21-4f11-bf31-ea0979421f9e" containerName="payload-processing" containerID="cri-o://6fd9d7b18b422921abf0ab26e7af42c1ac8e9535ba476e8d56f1adf7eb2b3ad7" gracePeriod=25
Oct 31 01:13:54.308377 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:54.308333    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z" podUID="9f274877-9424-47bc-9e10-5df2c2455a4b" containerName="payload-pre-processing" containerID="cri-o://41ca7d2600aeeccf36fd61d010f29e250febd49b516c4e5d2a01f251bcb64220" gracePeriod=25
Oct 31 01:13:54.617707 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:54.617661    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d"
Oct 31 01:13:54.697816 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:54.697779    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z"
Oct 31 01:13:54.713218 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:54.713187    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d7550dd-2f21-4f11-bf31-ea0979421f9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d7550dd-2f21-4f11-bf31-ea0979421f9e-plugins-config-volume\") pod \"6d7550dd-2f21-4f11-bf31-ea0979421f9e\" (UID: \"6d7550dd-2f21-4f11-bf31-ea0979421f9e\") "
Oct 31 01:13:54.713499 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:54.713479    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d7550dd-2f21-4f11-bf31-ea0979421f9e-kube-api-access-2bzk9\" (UniqueName: \"kubernetes.io/projected/6d7550dd-2f21-4f11-bf31-ea0979421f9e-kube-api-access-2bzk9\") pod \"6d7550dd-2f21-4f11-bf31-ea0979421f9e\" (UID: \"6d7550dd-2f21-4f11-bf31-ea0979421f9e\") "
Oct 31 01:13:54.713662 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:54.713626    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d7550dd-2f21-4f11-bf31-ea0979421f9e-plugins-config-volume" pod "6d7550dd-2f21-4f11-bf31-ea0979421f9e" (UID: "6d7550dd-2f21-4f11-bf31-ea0979421f9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:13:54.714071 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:54.714050    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d7550dd-2f21-4f11-bf31-ea0979421f9e-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:13:54.716150 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:54.716126    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d7550dd-2f21-4f11-bf31-ea0979421f9e-kube-api-access-2bzk9" pod "6d7550dd-2f21-4f11-bf31-ea0979421f9e" (UID: "6d7550dd-2f21-4f11-bf31-ea0979421f9e"). InnerVolumeSpecName "kube-api-access-2bzk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:13:54.814890 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:54.814856    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f274877-9424-47bc-9e10-5df2c2455a4b-kube-api-access-6swbw\" (UniqueName: \"kubernetes.io/projected/9f274877-9424-47bc-9e10-5df2c2455a4b-kube-api-access-6swbw\") pod \"9f274877-9424-47bc-9e10-5df2c2455a4b\" (UID: \"9f274877-9424-47bc-9e10-5df2c2455a4b\") "
Oct 31 01:13:54.815050 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:54.814999    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f274877-9424-47bc-9e10-5df2c2455a4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f274877-9424-47bc-9e10-5df2c2455a4b-plugins-config-volume\") pod \"9f274877-9424-47bc-9e10-5df2c2455a4b\" (UID: \"9f274877-9424-47bc-9e10-5df2c2455a4b\") "
Oct 31 01:13:54.815433 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:54.815402    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f274877-9424-47bc-9e10-5df2c2455a4b-plugins-config-volume" pod "9f274877-9424-47bc-9e10-5df2c2455a4b" (UID: "9f274877-9424-47bc-9e10-5df2c2455a4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:13:54.815719 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:54.815657    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f274877-9424-47bc-9e10-5df2c2455a4b-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:13:54.815719 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:54.815700    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bzk9\" (UniqueName: \"kubernetes.io/projected/6d7550dd-2f21-4f11-bf31-ea0979421f9e-kube-api-access-2bzk9\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:13:54.817385 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:54.817305    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f274877-9424-47bc-9e10-5df2c2455a4b-kube-api-access-6swbw" pod "9f274877-9424-47bc-9e10-5df2c2455a4b" (UID: "9f274877-9424-47bc-9e10-5df2c2455a4b"). InnerVolumeSpecName "kube-api-access-6swbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:13:54.916946 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:54.916907    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6swbw\" (UniqueName: \"kubernetes.io/projected/9f274877-9424-47bc-9e10-5df2c2455a4b-kube-api-access-6swbw\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:13:55.165805 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:55.165765    4327 generic.go:356] "Generic (PLEG): container finished" podID="9f274877-9424-47bc-9e10-5df2c2455a4b" containerID="41ca7d2600aeeccf36fd61d010f29e250febd49b516c4e5d2a01f251bcb64220" exitCode=0
Oct 31 01:13:55.165993 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:55.165823    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z"
Oct 31 01:13:55.165993 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:55.165859    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z" event={"ID":"9f274877-9424-47bc-9e10-5df2c2455a4b","Type":"ContainerDied","Data":"41ca7d2600aeeccf36fd61d010f29e250febd49b516c4e5d2a01f251bcb64220"}
Oct 31 01:13:55.165993 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:55.165920    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z" event={"ID":"9f274877-9424-47bc-9e10-5df2c2455a4b","Type":"ContainerDied","Data":"884dc118f70f2ca971303aa12546996898e77a8d8e301770e81037ab4891c9d6"}
Oct 31 01:13:55.165993 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:55.165946    4327 scope.go:122] "RemoveContainer" containerID="41ca7d2600aeeccf36fd61d010f29e250febd49b516c4e5d2a01f251bcb64220"
Oct 31 01:13:55.167488 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:55.167458    4327 generic.go:356] "Generic (PLEG): container finished" podID="6d7550dd-2f21-4f11-bf31-ea0979421f9e" containerID="6fd9d7b18b422921abf0ab26e7af42c1ac8e9535ba476e8d56f1adf7eb2b3ad7" exitCode=0
Oct 31 01:13:55.167618 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:55.167505    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d"
Oct 31 01:13:55.167618 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:55.167498    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d" event={"ID":"6d7550dd-2f21-4f11-bf31-ea0979421f9e","Type":"ContainerDied","Data":"6fd9d7b18b422921abf0ab26e7af42c1ac8e9535ba476e8d56f1adf7eb2b3ad7"}
Oct 31 01:13:55.167769 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:55.167641    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d" event={"ID":"6d7550dd-2f21-4f11-bf31-ea0979421f9e","Type":"ContainerDied","Data":"ab1a5da2babb45ec8191d2e74cf856414315a4de4a243b06fefc0f9c638d98a2"}
Oct 31 01:13:55.185731 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:55.185712    4327 scope.go:122] "RemoveContainer" containerID="41ca7d2600aeeccf36fd61d010f29e250febd49b516c4e5d2a01f251bcb64220"
Oct 31 01:13:55.186127 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:13:55.186094    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41ca7d2600aeeccf36fd61d010f29e250febd49b516c4e5d2a01f251bcb64220\": container with ID starting with 41ca7d2600aeeccf36fd61d010f29e250febd49b516c4e5d2a01f251bcb64220 not found: ID does not exist" containerID="41ca7d2600aeeccf36fd61d010f29e250febd49b516c4e5d2a01f251bcb64220"
Oct 31 01:13:55.186241 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:55.186136    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41ca7d2600aeeccf36fd61d010f29e250febd49b516c4e5d2a01f251bcb64220"} err="failed to get container status \"41ca7d2600aeeccf36fd61d010f29e250febd49b516c4e5d2a01f251bcb64220\": rpc error: code = NotFound desc = could not find container \"41ca7d2600aeeccf36fd61d010f29e250febd49b516c4e5d2a01f251bcb64220\": container with ID starting with 41ca7d2600aeeccf36fd61d010f29e250febd49b516c4e5d2a01f251bcb64220 not found: ID does not exist"
Oct 31 01:13:55.186241 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:55.186162    4327 scope.go:122] "RemoveContainer" containerID="6fd9d7b18b422921abf0ab26e7af42c1ac8e9535ba476e8d56f1adf7eb2b3ad7"
Oct 31 01:13:55.196414 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:55.196391    4327 scope.go:122] "RemoveContainer" containerID="6fd9d7b18b422921abf0ab26e7af42c1ac8e9535ba476e8d56f1adf7eb2b3ad7"
Oct 31 01:13:55.196736 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:13:55.196706    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fd9d7b18b422921abf0ab26e7af42c1ac8e9535ba476e8d56f1adf7eb2b3ad7\": container with ID starting with 6fd9d7b18b422921abf0ab26e7af42c1ac8e9535ba476e8d56f1adf7eb2b3ad7 not found: ID does not exist" containerID="6fd9d7b18b422921abf0ab26e7af42c1ac8e9535ba476e8d56f1adf7eb2b3ad7"
Oct 31 01:13:55.196839 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:55.196746    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fd9d7b18b422921abf0ab26e7af42c1ac8e9535ba476e8d56f1adf7eb2b3ad7"} err="failed to get container status \"6fd9d7b18b422921abf0ab26e7af42c1ac8e9535ba476e8d56f1adf7eb2b3ad7\": rpc error: code = NotFound desc = could not find container \"6fd9d7b18b422921abf0ab26e7af42c1ac8e9535ba476e8d56f1adf7eb2b3ad7\": container with ID starting with 6fd9d7b18b422921abf0ab26e7af42c1ac8e9535ba476e8d56f1adf7eb2b3ad7 not found: ID does not exist"
Oct 31 01:13:55.224565 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:55.224528    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z"]
Oct 31 01:13:55.234506 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:55.234476    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-6686c749f5-cbh5z"]
Oct 31 01:13:55.261801 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:55.261766    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d"]
Oct 31 01:13:55.266084 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:55.266045    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e41918-7f99fb8c89-7lq8d"]
Oct 31 01:13:55.922943 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:55.922913    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d7550dd-2f21-4f11-bf31-ea0979421f9e" path="/var/lib/kubelet/pods/6d7550dd-2f21-4f11-bf31-ea0979421f9e/volumes"
Oct 31 01:13:55.923429 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:55.923414    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f274877-9424-47bc-9e10-5df2c2455a4b" path="/var/lib/kubelet/pods/9f274877-9424-47bc-9e10-5df2c2455a4b/volumes"
Oct 31 01:13:57.467003 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.466972    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j"]
Oct 31 01:13:57.467709 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.467690    4327 cpu_manager.go:414] "containerMap: removing container" podUID="9f274877-9424-47bc-9e10-5df2c2455a4b" containerName="payload-pre-processing"
Oct 31 01:13:57.467769 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.467707    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f274877-9424-47bc-9e10-5df2c2455a4b" containerName="payload-pre-processing"
Oct 31 01:13:57.467769 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.467751    4327 cpu_manager.go:414] "containerMap: removing container" podUID="6d7550dd-2f21-4f11-bf31-ea0979421f9e" containerName="payload-processing"
Oct 31 01:13:57.467769 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.467757    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d7550dd-2f21-4f11-bf31-ea0979421f9e" containerName="payload-processing"
Oct 31 01:13:57.467881 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.467867    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="d93e1303-7225-4f18-b344-ffeea15da3b5" containerName="istio-proxy"
Oct 31 01:13:57.467915 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.467882    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d7550dd-2f21-4f11-bf31-ea0979421f9e" containerName="payload-processing"
Oct 31 01:13:57.467915 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.467893    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f274877-9424-47bc-9e10-5df2c2455a4b" containerName="payload-pre-processing"
Oct 31 01:13:57.473048 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.473024    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j"
Oct 31 01:13:57.487785 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.487760    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh"]
Oct 31 01:13:57.493132 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.493105    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh"
Oct 31 01:13:57.494363 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.494320    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j"]
Oct 31 01:13:57.496087 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.496062    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-33d44f\""
Oct 31 01:13:57.496225 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.496154    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-33d44f-dockercfg-mmdmq\""
Oct 31 01:13:57.520383 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.520350    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw"]
Oct 31 01:13:57.530377 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.530341    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh"]
Oct 31 01:13:57.530555 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.530457    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw"
Oct 31 01:13:57.547119 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.547082    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw"]
Oct 31 01:13:57.592327 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.592296    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-4207cd-serving-cert\""
Oct 31 01:13:57.653931 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.653898    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-metrics-certs\") pod \"maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh\" (UID: \"0280fb12-2007-4c5c-89ca-bf2f18dc28ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh"
Oct 31 01:13:57.654139 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.653973    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwbdq\" (UniqueName: \"kubernetes.io/projected/7aaf0c45-ac88-4c0f-b2a4-ab5539740af6-kube-api-access-pwbdq\") pod \"payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw\" (UID: \"7aaf0c45-ac88-4c0f-b2a4-ab5539740af6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw"
Oct 31 01:13:57.654139 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.654027    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r7lc\" (UniqueName: \"kubernetes.io/projected/811136df-a3ab-4a3f-b826-8e7aea058eb8-kube-api-access-7r7lc\") pod \"payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j\" (UID: \"811136df-a3ab-4a3f-b826-8e7aea058eb8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j"
Oct 31 01:13:57.654139 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.654109    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/811136df-a3ab-4a3f-b826-8e7aea058eb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j\" (UID: \"811136df-a3ab-4a3f-b826-8e7aea058eb8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j"
Oct 31 01:13:57.654139 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.654131    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh\" (UID: \"0280fb12-2007-4c5c-89ca-bf2f18dc28ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh"
Oct 31 01:13:57.654379 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.654317    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aaf0c45-ac88-4c0f-b2a4-ab5539740af6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw\" (UID: \"7aaf0c45-ac88-4c0f-b2a4-ab5539740af6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw"
Oct 31 01:13:57.654471 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.654451    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmxzt\" (UniqueName: \"kubernetes.io/projected/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-kube-api-access-kmxzt\") pod \"maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh\" (UID: \"0280fb12-2007-4c5c-89ca-bf2f18dc28ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh"
Oct 31 01:13:57.654589 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.654574    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh\" (UID: \"0280fb12-2007-4c5c-89ca-bf2f18dc28ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh"
Oct 31 01:13:57.758721 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.758601    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmxzt\" (UniqueName: \"kubernetes.io/projected/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-kube-api-access-kmxzt\") pod \"maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh\" (UID: \"0280fb12-2007-4c5c-89ca-bf2f18dc28ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh"
Oct 31 01:13:57.759448 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.759419    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh\" (UID: \"0280fb12-2007-4c5c-89ca-bf2f18dc28ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh"
Oct 31 01:13:57.760447 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.760426    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-metrics-certs\") pod \"maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh\" (UID: \"0280fb12-2007-4c5c-89ca-bf2f18dc28ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh"
Oct 31 01:13:57.761124 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.761105    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwbdq\" (UniqueName: \"kubernetes.io/projected/7aaf0c45-ac88-4c0f-b2a4-ab5539740af6-kube-api-access-pwbdq\") pod \"payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw\" (UID: \"7aaf0c45-ac88-4c0f-b2a4-ab5539740af6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw"
Oct 31 01:13:57.761594 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.761569    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r7lc\" (UniqueName: \"kubernetes.io/projected/811136df-a3ab-4a3f-b826-8e7aea058eb8-kube-api-access-7r7lc\") pod \"payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j\" (UID: \"811136df-a3ab-4a3f-b826-8e7aea058eb8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j"
Oct 31 01:13:57.761738 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.761613    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/811136df-a3ab-4a3f-b826-8e7aea058eb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j\" (UID: \"811136df-a3ab-4a3f-b826-8e7aea058eb8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j"
Oct 31 01:13:57.761738 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.761643    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh\" (UID: \"0280fb12-2007-4c5c-89ca-bf2f18dc28ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh"
Oct 31 01:13:57.766047 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.760346    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh\" (UID: \"0280fb12-2007-4c5c-89ca-bf2f18dc28ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh"
Oct 31 01:13:57.766351 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.766326    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aaf0c45-ac88-4c0f-b2a4-ab5539740af6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw\" (UID: \"7aaf0c45-ac88-4c0f-b2a4-ab5539740af6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw"
Oct 31 01:13:57.771547 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.767394    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aaf0c45-ac88-4c0f-b2a4-ab5539740af6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw\" (UID: \"7aaf0c45-ac88-4c0f-b2a4-ab5539740af6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw"
Oct 31 01:13:57.771547 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.768639    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/811136df-a3ab-4a3f-b826-8e7aea058eb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j\" (UID: \"811136df-a3ab-4a3f-b826-8e7aea058eb8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j"
Oct 31 01:13:57.771547 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:13:57.768754    4327 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-serving-cert: secret "maas-api-e2e-ipp-a-4207cd-serving-cert" not found
Oct 31 01:13:57.771547 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:13:57.768843    4327 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-maas-api-tls podName:0280fb12-2007-4c5c-89ca-bf2f18dc28ad nodeName:}" failed. No retries permitted until 2026-10-31 01:13:58.268823256 +0000 UTC m=+3298.490858789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-maas-api-tls") pod "maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh" (UID: "0280fb12-2007-4c5c-89ca-bf2f18dc28ad") : secret "maas-api-e2e-ipp-a-4207cd-serving-cert" not found
Oct 31 01:13:57.774377 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.774178    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-metrics-certs\") pod \"maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh\" (UID: \"0280fb12-2007-4c5c-89ca-bf2f18dc28ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh"
Oct 31 01:13:57.800738 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.796091    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r7lc\" (UniqueName: \"kubernetes.io/projected/811136df-a3ab-4a3f-b826-8e7aea058eb8-kube-api-access-7r7lc\") pod \"payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j\" (UID: \"811136df-a3ab-4a3f-b826-8e7aea058eb8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j"
Oct 31 01:13:57.805492 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.805458    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmxzt\" (UniqueName: \"kubernetes.io/projected/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-kube-api-access-kmxzt\") pod \"maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh\" (UID: \"0280fb12-2007-4c5c-89ca-bf2f18dc28ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh"
Oct 31 01:13:57.807058 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.807029    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwbdq\" (UniqueName: \"kubernetes.io/projected/7aaf0c45-ac88-4c0f-b2a4-ab5539740af6-kube-api-access-pwbdq\") pod \"payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw\" (UID: \"7aaf0c45-ac88-4c0f-b2a4-ab5539740af6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw"
Oct 31 01:13:57.865223 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:57.865190    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw"
Oct 31 01:13:58.089764 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:58.089662    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j"
Oct 31 01:13:58.181749 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:13:58.181713    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aaf0c45_ac88_4c0f_b2a4_ab5539740af6.slice/crio-aec780df60d2c44ed19e7eb617a5c0c106d92313a9a587e013714221313486ff WatchSource:0}: Error finding container aec780df60d2c44ed19e7eb617a5c0c106d92313a9a587e013714221313486ff: Status 404 returned error can't find the container with id aec780df60d2c44ed19e7eb617a5c0c106d92313a9a587e013714221313486ff
Oct 31 01:13:58.182953 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:58.182926    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw"]
Oct 31 01:13:58.193449 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:58.193405    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw" event={"ID":"7aaf0c45-ac88-4c0f-b2a4-ab5539740af6","Type":"ContainerStarted","Data":"aec780df60d2c44ed19e7eb617a5c0c106d92313a9a587e013714221313486ff"}
Oct 31 01:13:58.273986 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:58.273955    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh\" (UID: \"0280fb12-2007-4c5c-89ca-bf2f18dc28ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh"
Oct 31 01:13:58.276245 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:58.276223    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh\" (UID: \"0280fb12-2007-4c5c-89ca-bf2f18dc28ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh"
Oct 31 01:13:58.395965 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:13:58.395916    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod811136df_a3ab_4a3f_b826_8e7aea058eb8.slice/crio-b35948f71875fa33b2c63cf73e75e06896be572bdf3beb9612b1455d9537f17c WatchSource:0}: Error finding container b35948f71875fa33b2c63cf73e75e06896be572bdf3beb9612b1455d9537f17c: Status 404 returned error can't find the container with id b35948f71875fa33b2c63cf73e75e06896be572bdf3beb9612b1455d9537f17c
Oct 31 01:13:58.398414 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:58.398387    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j"]
Oct 31 01:13:58.414534 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:58.414506    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh"
Oct 31 01:13:58.767493 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:58.767463    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh"]
Oct 31 01:13:59.201870 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:59.201831    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh" event={"ID":"0280fb12-2007-4c5c-89ca-bf2f18dc28ad","Type":"ContainerStarted","Data":"544d3f6b205769448adcb7be5f4171264e7039b38423a4167920c34b4311bc22"}
Oct 31 01:13:59.203576 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:59.203544    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw" event={"ID":"7aaf0c45-ac88-4c0f-b2a4-ab5539740af6","Type":"ContainerStarted","Data":"a57932f6496a31b07617c27fede5614469cb6c10ee5f0ad3a0fda64bdb28e1b2"}
Oct 31 01:13:59.203771 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:59.203627    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw"
Oct 31 01:13:59.205320 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:59.205293    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j" event={"ID":"811136df-a3ab-4a3f-b826-8e7aea058eb8","Type":"ContainerStarted","Data":"51de4b0d5c1331177448422b586381d4c3d8527786ad33d2d0658207aedfa169"}
Oct 31 01:13:59.205449 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:59.205339    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j" event={"ID":"811136df-a3ab-4a3f-b826-8e7aea058eb8","Type":"ContainerStarted","Data":"b35948f71875fa33b2c63cf73e75e06896be572bdf3beb9612b1455d9537f17c"}
Oct 31 01:13:59.205449 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:59.205374    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j"
Oct 31 01:13:59.248727 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:59.248638    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw" podStartSLOduration=2.248620031 podStartE2EDuration="2.248620031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:59.245511341 +0000 UTC m=+3299.467546884" watchObservedRunningTime="2026-10-31 01:13:59.248620031 +0000 UTC m=+3299.470655573"
Oct 31 01:13:59.293911 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:13:59.293865    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j" podStartSLOduration=2.293850354 podStartE2EDuration="2.293850354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:59.292743234 +0000 UTC m=+3299.514778780" watchObservedRunningTime="2026-10-31 01:13:59.293850354 +0000 UTC m=+3299.515885945"
Oct 31 01:14:00.214910 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:00.214869    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh" event={"ID":"0280fb12-2007-4c5c-89ca-bf2f18dc28ad","Type":"ContainerStarted","Data":"40bdab51d128b374918138955f1c30b420938a057ec79ca589171aeb4a6b335f"}
Oct 31 01:14:00.215396 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:00.214971    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh"
Oct 31 01:14:00.260254 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:00.260202    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh" podStartSLOduration=2.816661442 podStartE2EDuration="3.260188598s" totalImagesPullingTime="443.527156ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:00.258178997 +0000 UTC m=+3300.480214539" watchObservedRunningTime="2026-10-31 01:14:00.260188598 +0000 UTC m=+3300.482224179"
Oct 31 01:14:06.234039 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:06.234005    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh"
Oct 31 01:14:09.156899 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:09.156867    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"]
Oct 31 01:14:09.157350 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:09.157089    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j" podUID="ee82474b-1ba2-4ade-8db5-161af9e5f3a4" containerName="istio-proxy" containerID="cri-o://d3663e71438b0464804dc5e7b9c7be074cdf36a8b6d5c4a25bf85b623763ec5f" gracePeriod=30
Oct 31 01:14:10.217868 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:10.217832    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw"
Oct 31 01:14:10.218347 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:10.217921    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j"
Oct 31 01:14:14.362999 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.362957    4327 generic.go:356] "Generic (PLEG): container finished" podID="ee82474b-1ba2-4ade-8db5-161af9e5f3a4" containerID="d3663e71438b0464804dc5e7b9c7be074cdf36a8b6d5c4a25bf85b623763ec5f" exitCode=0
Oct 31 01:14:14.363360 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.363124    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j" event={"ID":"ee82474b-1ba2-4ade-8db5-161af9e5f3a4","Type":"ContainerDied","Data":"d3663e71438b0464804dc5e7b9c7be074cdf36a8b6d5c4a25bf85b623763ec5f"}
Oct 31 01:14:14.363360 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.363187    4327 scope.go:122] "RemoveContainer" containerID="41695b1410ea58459f9f2133cd6c8f2ea38b45fcefdea9b4eb1b04b8b38e1f75"
Oct 31 01:14:14.447226 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.447167    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:14:14.453866 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.453839    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-podinfo\") pod \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") "
Oct 31 01:14:14.454032 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.453930    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-token\" (UniqueName: \"kubernetes.io/projected/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-token\") pod \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") "
Oct 31 01:14:14.454032 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.453971    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istiod-ca-cert\") pod \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") "
Oct 31 01:14:14.454032 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.454017    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-credential-socket\") pod \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") "
Oct 31 01:14:14.454202 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.454044    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-workload-certs\") pod \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") "
Oct 31 01:14:14.454202 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.454120    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-kube-api-access-rrpg7\" (UniqueName: \"kubernetes.io/projected/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-kube-api-access-rrpg7\") pod \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") "
Oct 31 01:14:14.454312 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.454214    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-workload-socket\") pod \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") "
Oct 31 01:14:14.454370 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.454299    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-credential-socket" pod "ee82474b-1ba2-4ade-8db5-161af9e5f3a4" (UID: "ee82474b-1ba2-4ade-8db5-161af9e5f3a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:14.454370 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.454320    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-data\") pod \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") "
Oct 31 01:14:14.454370 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.454355    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-envoy\") pod \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\" (UID: \"ee82474b-1ba2-4ade-8db5-161af9e5f3a4\") "
Oct 31 01:14:14.454533 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.454417    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istiod-ca-cert" pod "ee82474b-1ba2-4ade-8db5-161af9e5f3a4" (UID: "ee82474b-1ba2-4ade-8db5-161af9e5f3a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:14:14.454617 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.454594    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-data" pod "ee82474b-1ba2-4ade-8db5-161af9e5f3a4" (UID: "ee82474b-1ba2-4ade-8db5-161af9e5f3a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:14.454776 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.454750    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-workload-socket" pod "ee82474b-1ba2-4ade-8db5-161af9e5f3a4" (UID: "ee82474b-1ba2-4ade-8db5-161af9e5f3a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:14.455006 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.454979    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-workload-certs" pod "ee82474b-1ba2-4ade-8db5-161af9e5f3a4" (UID: "ee82474b-1ba2-4ade-8db5-161af9e5f3a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:14.455187 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.455106    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:14:14.455187 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.455129    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:14:14.455187 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.455143    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:14:14.455187 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.455156    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:14:14.455187 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.455168    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:14:14.456472 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.456432    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-podinfo" pod "ee82474b-1ba2-4ade-8db5-161af9e5f3a4" (UID: "ee82474b-1ba2-4ade-8db5-161af9e5f3a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:14:14.456712 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.456666    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-token" pod "ee82474b-1ba2-4ade-8db5-161af9e5f3a4" (UID: "ee82474b-1ba2-4ade-8db5-161af9e5f3a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:14:14.456790 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.456752    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-envoy" pod "ee82474b-1ba2-4ade-8db5-161af9e5f3a4" (UID: "ee82474b-1ba2-4ade-8db5-161af9e5f3a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:14.456966 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.456948    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-kube-api-access-rrpg7" pod "ee82474b-1ba2-4ade-8db5-161af9e5f3a4" (UID: "ee82474b-1ba2-4ade-8db5-161af9e5f3a4"). InnerVolumeSpecName "kube-api-access-rrpg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:14:14.556160 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.556129    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:14:14.556160 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.556158    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:14:14.556160 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.556167    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:14:14.556398 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:14.556176    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrpg7\" (UniqueName: \"kubernetes.io/projected/ee82474b-1ba2-4ade-8db5-161af9e5f3a4-kube-api-access-rrpg7\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:14:15.294018 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:14:15.293978    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:14:15.371290 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:15.371257    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"
Oct 31 01:14:15.371830 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:15.371295    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j" event={"ID":"ee82474b-1ba2-4ade-8db5-161af9e5f3a4","Type":"ContainerDied","Data":"8ba7bfc640445b2ce652c31431b2cd119c72a40bc2276d802bcd5be860044c14"}
Oct 31 01:14:15.371830 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:15.371357    4327 scope.go:122] "RemoveContainer" containerID="d3663e71438b0464804dc5e7b9c7be074cdf36a8b6d5c4a25bf85b623763ec5f"
Oct 31 01:14:15.440284 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:15.440250    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"]
Oct 31 01:14:15.452691 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:15.452650    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-73f915-openshift-default-64b6f4c569-qq92j"]
Oct 31 01:14:15.922859 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:15.922828    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee82474b-1ba2-4ade-8db5-161af9e5f3a4" path="/var/lib/kubelet/pods/ee82474b-1ba2-4ade-8db5-161af9e5f3a4/volumes"
Oct 31 01:14:25.476124 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.476091    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h"]
Oct 31 01:14:25.478003 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.477973    4327 cpu_manager.go:414] "containerMap: removing container" podUID="ee82474b-1ba2-4ade-8db5-161af9e5f3a4" containerName="istio-proxy"
Oct 31 01:14:25.478194 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.478176    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee82474b-1ba2-4ade-8db5-161af9e5f3a4" containerName="istio-proxy"
Oct 31 01:14:25.478332 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.478310    4327 cpu_manager.go:414] "containerMap: removing container" podUID="ee82474b-1ba2-4ade-8db5-161af9e5f3a4" containerName="istio-proxy"
Oct 31 01:14:25.478332 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.478327    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee82474b-1ba2-4ade-8db5-161af9e5f3a4" containerName="istio-proxy"
Oct 31 01:14:25.478531 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.478517    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee82474b-1ba2-4ade-8db5-161af9e5f3a4" containerName="istio-proxy"
Oct 31 01:14:25.478594 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.478540    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee82474b-1ba2-4ade-8db5-161af9e5f3a4" containerName="istio-proxy"
Oct 31 01:14:25.483403 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.483376    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h"
Oct 31 01:14:25.491894 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.491861    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h"]
Oct 31 01:14:25.502786 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.502002    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e41918\""
Oct 31 01:14:25.502786 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.502155    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e41918-dockercfg-h9fvk\""
Oct 31 01:14:25.543308 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.543270    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfjg2\" (UniqueName: \"kubernetes.io/projected/5c54b47b-3540-45b9-8ac7-8b1f056b0b81-kube-api-access-nfjg2\") pod \"payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h\" (UID: \"5c54b47b-3540-45b9-8ac7-8b1f056b0b81\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h"
Oct 31 01:14:25.543524 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.543499    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c54b47b-3540-45b9-8ac7-8b1f056b0b81-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h\" (UID: \"5c54b47b-3540-45b9-8ac7-8b1f056b0b81\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h"
Oct 31 01:14:25.644891 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.644844    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfjg2\" (UniqueName: \"kubernetes.io/projected/5c54b47b-3540-45b9-8ac7-8b1f056b0b81-kube-api-access-nfjg2\") pod \"payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h\" (UID: \"5c54b47b-3540-45b9-8ac7-8b1f056b0b81\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h"
Oct 31 01:14:25.645080 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.644993    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c54b47b-3540-45b9-8ac7-8b1f056b0b81-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h\" (UID: \"5c54b47b-3540-45b9-8ac7-8b1f056b0b81\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h"
Oct 31 01:14:25.645793 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.645770    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c54b47b-3540-45b9-8ac7-8b1f056b0b81-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h\" (UID: \"5c54b47b-3540-45b9-8ac7-8b1f056b0b81\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h"
Oct 31 01:14:25.654067 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.654034    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57"]
Oct 31 01:14:25.658982 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.658960    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57"
Oct 31 01:14:25.670866 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.670844    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfjg2\" (UniqueName: \"kubernetes.io/projected/5c54b47b-3540-45b9-8ac7-8b1f056b0b81-kube-api-access-nfjg2\") pod \"payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h\" (UID: \"5c54b47b-3540-45b9-8ac7-8b1f056b0b81\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h"
Oct 31 01:14:25.681812 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.681783    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57"]
Oct 31 01:14:25.746280 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.746198    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fmc6\" (UniqueName: \"kubernetes.io/projected/7900b962-7b7a-4d5e-b78f-8d8fe95d3123-kube-api-access-2fmc6\") pod \"payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57\" (UID: \"7900b962-7b7a-4d5e-b78f-8d8fe95d3123\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57"
Oct 31 01:14:25.746452 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.746381    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7900b962-7b7a-4d5e-b78f-8d8fe95d3123-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57\" (UID: \"7900b962-7b7a-4d5e-b78f-8d8fe95d3123\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57"
Oct 31 01:14:25.820330 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.820284    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h"
Oct 31 01:14:25.854567 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.854528    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fmc6\" (UniqueName: \"kubernetes.io/projected/7900b962-7b7a-4d5e-b78f-8d8fe95d3123-kube-api-access-2fmc6\") pod \"payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57\" (UID: \"7900b962-7b7a-4d5e-b78f-8d8fe95d3123\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57"
Oct 31 01:14:25.854940 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.854721    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7900b962-7b7a-4d5e-b78f-8d8fe95d3123-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57\" (UID: \"7900b962-7b7a-4d5e-b78f-8d8fe95d3123\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57"
Oct 31 01:14:25.855752 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.855724    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7900b962-7b7a-4d5e-b78f-8d8fe95d3123-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57\" (UID: \"7900b962-7b7a-4d5e-b78f-8d8fe95d3123\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57"
Oct 31 01:14:25.862335 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.862302    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"]
Oct 31 01:14:25.882389 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.882328    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fmc6\" (UniqueName: \"kubernetes.io/projected/7900b962-7b7a-4d5e-b78f-8d8fe95d3123-kube-api-access-2fmc6\") pod \"payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57\" (UID: \"7900b962-7b7a-4d5e-b78f-8d8fe95d3123\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57"
Oct 31 01:14:25.980506 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:25.974914    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57"
Oct 31 01:14:26.157312 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:26.157256    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h"]
Oct 31 01:14:26.500419 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:26.500357    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57"]
Oct 31 01:14:26.521238 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:26.519295    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h" event={"ID":"5c54b47b-3540-45b9-8ac7-8b1f056b0b81","Type":"ContainerStarted","Data":"ea13a63ccd5acd13266ebf8430f643cee616cd2a3cd83fb2e3a89e5b62a80297"}
Oct 31 01:14:26.521238 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:26.519364    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h" event={"ID":"5c54b47b-3540-45b9-8ac7-8b1f056b0b81","Type":"ContainerStarted","Data":"c5a7bc6aee9b7c3fe8e9364995fbd5e95bb3f2579ed5fb3f95a6d975c4af9973"}
Oct 31 01:14:26.532066 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:26.531942    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h"
Oct 31 01:14:26.569831 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:26.569772    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h" podStartSLOduration=1.569756651 podStartE2EDuration="1.569756651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:26.565143408 +0000 UTC m=+3326.787178950" watchObservedRunningTime="2026-10-31 01:14:26.569756651 +0000 UTC m=+3326.791792192"
Oct 31 01:14:27.573746 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:27.568771    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57" event={"ID":"7900b962-7b7a-4d5e-b78f-8d8fe95d3123","Type":"ContainerStarted","Data":"820ae26d7dfeb0a7438f67557585735e4d601a74434620dfb6628a48a7112a25"}
Oct 31 01:14:27.573746 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:27.568831    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57" event={"ID":"7900b962-7b7a-4d5e-b78f-8d8fe95d3123","Type":"ContainerStarted","Data":"3f8c4bda9c599036082e14084f10c3b67c2a4e128d90fe6f41674b55e75f8db1"}
Oct 31 01:14:27.573746 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:27.568883    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57"
Oct 31 01:14:27.617866 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:27.617818    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57" podStartSLOduration=2.617803602 podStartE2EDuration="2.617803602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:27.611891685 +0000 UTC m=+3327.833927226" watchObservedRunningTime="2026-10-31 01:14:27.617803602 +0000 UTC m=+3327.839839173"
Oct 31 01:14:38.577243 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:38.577212    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h"
Oct 31 01:14:39.583802 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:39.583758    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57"
Oct 31 01:14:40.898772 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:40.898725    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8" podUID="1d2ed373-a68e-4048-a3d4-e3bcee6f290d" containerName="main" containerID="cri-o://02cf95672e17830b978fee45e8adc3d643bca03157876057122d15315d0f5b05" gracePeriod=45
Oct 31 01:14:41.182411 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.182383    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"
Oct 31 01:14:41.324194 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.324163    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-tmp-dir\") pod \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\" (UID: \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\") "
Oct 31 01:14:41.324510 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.324206    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-kube-api-access-682ws\" (UniqueName: \"kubernetes.io/projected/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-kube-api-access-682ws\") pod \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\" (UID: \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\") "
Oct 31 01:14:41.324510 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.324251    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-home\" (UniqueName: \"kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-home\") pod \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\" (UID: \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\") "
Oct 31 01:14:41.324510 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.324268    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-tls-certs\" (UniqueName: \"kubernetes.io/secret/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-tls-certs\") pod \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\" (UID: \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\") "
Oct 31 01:14:41.324510 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.324357    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-dshm\") pod \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\" (UID: \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\") "
Oct 31 01:14:41.324510 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.324431    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-tmp-dir" pod "1d2ed373-a68e-4048-a3d4-e3bcee6f290d" (UID: "1d2ed373-a68e-4048-a3d4-e3bcee6f290d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:41.324510 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.324447    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-model-cache\") pod \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\" (UID: \"1d2ed373-a68e-4048-a3d4-e3bcee6f290d\") "
Oct 31 01:14:41.324882 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.324520    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-home" pod "1d2ed373-a68e-4048-a3d4-e3bcee6f290d" (UID: "1d2ed373-a68e-4048-a3d4-e3bcee6f290d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:41.324882 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.324730    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-model-cache" pod "1d2ed373-a68e-4048-a3d4-e3bcee6f290d" (UID: "1d2ed373-a68e-4048-a3d4-e3bcee6f290d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:41.325002 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.324989    4327 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-home\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:14:41.325068 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.325005    4327 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-model-cache\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:14:41.325068 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.325018    4327 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-tmp-dir\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:14:41.326659 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.326637    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-tls-certs" pod "1d2ed373-a68e-4048-a3d4-e3bcee6f290d" (UID: "1d2ed373-a68e-4048-a3d4-e3bcee6f290d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:14:41.326799 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.326720    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-kube-api-access-682ws" pod "1d2ed373-a68e-4048-a3d4-e3bcee6f290d" (UID: "1d2ed373-a68e-4048-a3d4-e3bcee6f290d"). InnerVolumeSpecName "kube-api-access-682ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:14:41.326799 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.326758    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-dshm" pod "1d2ed373-a68e-4048-a3d4-e3bcee6f290d" (UID: "1d2ed373-a68e-4048-a3d4-e3bcee6f290d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:41.426143 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.426054    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-682ws\" (UniqueName: \"kubernetes.io/projected/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-kube-api-access-682ws\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:14:41.426143 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.426084    4327 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-tls-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:14:41.426143 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.426094    4327 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d2ed373-a68e-4048-a3d4-e3bcee6f290d-dshm\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:14:41.678312 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.678227    4327 generic.go:356] "Generic (PLEG): container finished" podID="1d2ed373-a68e-4048-a3d4-e3bcee6f290d" containerID="02cf95672e17830b978fee45e8adc3d643bca03157876057122d15315d0f5b05" exitCode=0
Oct 31 01:14:41.678312 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.678270    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"
Oct 31 01:14:41.678542 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.678312    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8" event={"ID":"1d2ed373-a68e-4048-a3d4-e3bcee6f290d","Type":"ContainerDied","Data":"02cf95672e17830b978fee45e8adc3d643bca03157876057122d15315d0f5b05"}
Oct 31 01:14:41.678542 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.678359    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8" event={"ID":"1d2ed373-a68e-4048-a3d4-e3bcee6f290d","Type":"ContainerDied","Data":"54da1b58b0ff6511c0709b74e92c0e38522329deec162df8475822c046bbff88"}
Oct 31 01:14:41.678542 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.678378    4327 scope.go:122] "RemoveContainer" containerID="02cf95672e17830b978fee45e8adc3d643bca03157876057122d15315d0f5b05"
Oct 31 01:14:41.689724 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.689703    4327 scope.go:122] "RemoveContainer" containerID="02cf95672e17830b978fee45e8adc3d643bca03157876057122d15315d0f5b05"
Oct 31 01:14:41.690059 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:14:41.690028    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02cf95672e17830b978fee45e8adc3d643bca03157876057122d15315d0f5b05\": container with ID starting with 02cf95672e17830b978fee45e8adc3d643bca03157876057122d15315d0f5b05 not found: ID does not exist" containerID="02cf95672e17830b978fee45e8adc3d643bca03157876057122d15315d0f5b05"
Oct 31 01:14:41.690131 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.690073    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02cf95672e17830b978fee45e8adc3d643bca03157876057122d15315d0f5b05"} err="failed to get container status \"02cf95672e17830b978fee45e8adc3d643bca03157876057122d15315d0f5b05\": rpc error: code = NotFound desc = could not find container \"02cf95672e17830b978fee45e8adc3d643bca03157876057122d15315d0f5b05\": container with ID starting with 02cf95672e17830b978fee45e8adc3d643bca03157876057122d15315d0f5b05 not found: ID does not exist"
Oct 31 01:14:41.734973 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.734939    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"]
Oct 31 01:14:41.741670 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.741637    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-714f8f/rate-test-model-714f8f-kserve-6c686b7549-5gmk8"]
Oct 31 01:14:41.924286 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:41.924247    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d2ed373-a68e-4048-a3d4-e3bcee6f290d" path="/var/lib/kubelet/pods/1d2ed373-a68e-4048-a3d4-e3bcee6f290d/volumes"
Oct 31 01:14:44.420809 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:44.420775    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w"]
Oct 31 01:14:44.452780 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:44.452749    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc"]
Oct 31 01:14:45.296771 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:14:45.296733    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:14:49.421240 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:49.421187    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w" podUID="96158b4f-c939-403a-9e2f-c037ebc5cf0e" containerName="payload-processing" containerID="cri-o://cf063eb96950b0bc3b361f4c589df04e4b5549f6c45693c751c9dd5cce14c418" gracePeriod=25
Oct 31 01:14:49.453480 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:49.453425    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc" podUID="ede414eb-a19a-4753-9317-dcbbdca0ecf1" containerName="payload-pre-processing" containerID="cri-o://b394cee5815c523086c53b43993a7c66a7f6afa8f00d1c4c140c18422ca0d2dc" gracePeriod=25
Oct 31 01:14:49.743886 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:49.743854    4327 generic.go:356] "Generic (PLEG): container finished" podID="ede414eb-a19a-4753-9317-dcbbdca0ecf1" containerID="b394cee5815c523086c53b43993a7c66a7f6afa8f00d1c4c140c18422ca0d2dc" exitCode=0
Oct 31 01:14:49.744081 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:49.743925    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc" event={"ID":"ede414eb-a19a-4753-9317-dcbbdca0ecf1","Type":"ContainerDied","Data":"b394cee5815c523086c53b43993a7c66a7f6afa8f00d1c4c140c18422ca0d2dc"}
Oct 31 01:14:49.745584 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:49.745562    4327 generic.go:356] "Generic (PLEG): container finished" podID="96158b4f-c939-403a-9e2f-c037ebc5cf0e" containerID="cf063eb96950b0bc3b361f4c589df04e4b5549f6c45693c751c9dd5cce14c418" exitCode=0
Oct 31 01:14:49.745696 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:49.745632    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w" event={"ID":"96158b4f-c939-403a-9e2f-c037ebc5cf0e","Type":"ContainerDied","Data":"cf063eb96950b0bc3b361f4c589df04e4b5549f6c45693c751c9dd5cce14c418"}
Oct 31 01:14:49.745779 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:49.745701    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w" event={"ID":"96158b4f-c939-403a-9e2f-c037ebc5cf0e","Type":"ContainerDied","Data":"d7d926cd73b6fa2d52c45e471dc28a9fea32cca8ade4c382435a89947edd76c4"}
Oct 31 01:14:49.745779 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:49.745722    4327 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7d926cd73b6fa2d52c45e471dc28a9fea32cca8ade4c382435a89947edd76c4"
Oct 31 01:14:49.758431 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:49.758406    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w"
Oct 31 01:14:49.781346 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:49.781314    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc"
Oct 31 01:14:49.860250 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:49.860216    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96158b4f-c939-403a-9e2f-c037ebc5cf0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96158b4f-c939-403a-9e2f-c037ebc5cf0e-plugins-config-volume\") pod \"96158b4f-c939-403a-9e2f-c037ebc5cf0e\" (UID: \"96158b4f-c939-403a-9e2f-c037ebc5cf0e\") "
Oct 31 01:14:49.860449 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:49.860269    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede414eb-a19a-4753-9317-dcbbdca0ecf1-kube-api-access-jq5mb\" (UniqueName: \"kubernetes.io/projected/ede414eb-a19a-4753-9317-dcbbdca0ecf1-kube-api-access-jq5mb\") pod \"ede414eb-a19a-4753-9317-dcbbdca0ecf1\" (UID: \"ede414eb-a19a-4753-9317-dcbbdca0ecf1\") "
Oct 31 01:14:49.860449 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:49.860308    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ede414eb-a19a-4753-9317-dcbbdca0ecf1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ede414eb-a19a-4753-9317-dcbbdca0ecf1-plugins-config-volume\") pod \"ede414eb-a19a-4753-9317-dcbbdca0ecf1\" (UID: \"ede414eb-a19a-4753-9317-dcbbdca0ecf1\") "
Oct 31 01:14:49.860584 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:49.860535    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96158b4f-c939-403a-9e2f-c037ebc5cf0e-kube-api-access-22tpl\" (UniqueName: \"kubernetes.io/projected/96158b4f-c939-403a-9e2f-c037ebc5cf0e-kube-api-access-22tpl\") pod \"96158b4f-c939-403a-9e2f-c037ebc5cf0e\" (UID: \"96158b4f-c939-403a-9e2f-c037ebc5cf0e\") "
Oct 31 01:14:49.860740 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:49.860669    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96158b4f-c939-403a-9e2f-c037ebc5cf0e-plugins-config-volume" pod "96158b4f-c939-403a-9e2f-c037ebc5cf0e" (UID: "96158b4f-c939-403a-9e2f-c037ebc5cf0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:14:49.860865 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:49.860840    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede414eb-a19a-4753-9317-dcbbdca0ecf1-plugins-config-volume" pod "ede414eb-a19a-4753-9317-dcbbdca0ecf1" (UID: "ede414eb-a19a-4753-9317-dcbbdca0ecf1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:14:49.861358 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:49.861331    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96158b4f-c939-403a-9e2f-c037ebc5cf0e-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:14:49.861503 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:49.861363    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ede414eb-a19a-4753-9317-dcbbdca0ecf1-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:14:49.862964 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:49.862934    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede414eb-a19a-4753-9317-dcbbdca0ecf1-kube-api-access-jq5mb" pod "ede414eb-a19a-4753-9317-dcbbdca0ecf1" (UID: "ede414eb-a19a-4753-9317-dcbbdca0ecf1"). InnerVolumeSpecName "kube-api-access-jq5mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:14:49.863374 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:49.863355    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96158b4f-c939-403a-9e2f-c037ebc5cf0e-kube-api-access-22tpl" pod "96158b4f-c939-403a-9e2f-c037ebc5cf0e" (UID: "96158b4f-c939-403a-9e2f-c037ebc5cf0e"). InnerVolumeSpecName "kube-api-access-22tpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:14:49.963417 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:49.963326    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22tpl\" (UniqueName: \"kubernetes.io/projected/96158b4f-c939-403a-9e2f-c037ebc5cf0e-kube-api-access-22tpl\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:14:49.963417 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:49.963363    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq5mb\" (UniqueName: \"kubernetes.io/projected/ede414eb-a19a-4753-9317-dcbbdca0ecf1-kube-api-access-jq5mb\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:14:50.754192 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:50.754102    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc"
Oct 31 01:14:50.754192 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:50.754141    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc" event={"ID":"ede414eb-a19a-4753-9317-dcbbdca0ecf1","Type":"ContainerDied","Data":"151b8a4bc26a520c38902adbb77874e67b7486f918425a15dc2400f0b2fbfdda"}
Oct 31 01:14:50.754742 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:50.754214    4327 scope.go:122] "RemoveContainer" containerID="b394cee5815c523086c53b43993a7c66a7f6afa8f00d1c4c140c18422ca0d2dc"
Oct 31 01:14:50.754742 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:50.754311    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w"
Oct 31 01:14:50.858620 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:50.858583    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc"]
Oct 31 01:14:50.877271 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:50.877231    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-6dcf849dd7-nr6dc"]
Oct 31 01:14:50.909171 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:50.909136    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w"]
Oct 31 01:14:50.925129 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:50.925092    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4207cd-f974fcd45-22f7w"]
Oct 31 01:14:51.924332 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:51.924295    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96158b4f-c939-403a-9e2f-c037ebc5cf0e" path="/var/lib/kubelet/pods/96158b4f-c939-403a-9e2f-c037ebc5cf0e/volumes"
Oct 31 01:14:51.924919 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:14:51.924896    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ede414eb-a19a-4753-9317-dcbbdca0ecf1" path="/var/lib/kubelet/pods/ede414eb-a19a-4753-9317-dcbbdca0ecf1/volumes"
Oct 31 01:15:00.283135 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.283102    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-b5dbp"]
Oct 31 01:15:00.284297 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.284272    4327 cpu_manager.go:414] "containerMap: removing container" podUID="1d2ed373-a68e-4048-a3d4-e3bcee6f290d" containerName="main"
Oct 31 01:15:00.284297 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.284296    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d2ed373-a68e-4048-a3d4-e3bcee6f290d" containerName="main"
Oct 31 01:15:00.284482 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.284311    4327 cpu_manager.go:414] "containerMap: removing container" podUID="96158b4f-c939-403a-9e2f-c037ebc5cf0e" containerName="payload-processing"
Oct 31 01:15:00.284482 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.284318    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96158b4f-c939-403a-9e2f-c037ebc5cf0e" containerName="payload-processing"
Oct 31 01:15:00.284482 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.284331    4327 cpu_manager.go:414] "containerMap: removing container" podUID="ede414eb-a19a-4753-9317-dcbbdca0ecf1" containerName="payload-pre-processing"
Oct 31 01:15:00.284482 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.284339    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede414eb-a19a-4753-9317-dcbbdca0ecf1" containerName="payload-pre-processing"
Oct 31 01:15:00.284742 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.284554    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d2ed373-a68e-4048-a3d4-e3bcee6f290d" containerName="main"
Oct 31 01:15:00.284742 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.284570    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="96158b4f-c939-403a-9e2f-c037ebc5cf0e" containerName="payload-processing"
Oct 31 01:15:00.284742 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.284581    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede414eb-a19a-4753-9317-dcbbdca0ecf1" containerName="payload-pre-processing"
Oct 31 01:15:00.290196 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.290165    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-b5dbp"
Oct 31 01:15:00.303670 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.303631    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p"]
Oct 31 01:15:00.317180 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.317150    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p"
Oct 31 01:15:00.338277 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.338248    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-b5dbp"]
Oct 31 01:15:00.348414 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.346797    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p"]
Oct 31 01:15:00.351046 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.350850    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885"]
Oct 31 01:15:00.356579 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.356555    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885"
Oct 31 01:15:00.366079 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.366056    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl"]
Oct 31 01:15:00.372596 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.372570    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl"
Oct 31 01:15:00.373547 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.373522    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885"]
Oct 31 01:15:00.391872 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.391845    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl"]
Oct 31 01:15:00.396372 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.396349    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\""
Oct 31 01:15:00.414907 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.414849    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpwwb\" (UniqueName: \"kubernetes.io/projected/b5fdc538-6826-42ad-a662-5ed5ed9fc004-kube-api-access-qpwwb\") pod \"maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p\" (UID: \"b5fdc538-6826-42ad-a662-5ed5ed9fc004\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p"
Oct 31 01:15:00.415109 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.414947    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5fdc538-6826-42ad-a662-5ed5ed9fc004-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p\" (UID: \"b5fdc538-6826-42ad-a662-5ed5ed9fc004\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p"
Oct 31 01:15:00.415109 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.415034    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f3a0a27-f945-4566-9421-31d96a47f8f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885\" (UID: \"0f3a0a27-f945-4566-9421-31d96a47f8f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885"
Oct 31 01:15:00.415109 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.415070    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890155-b5dbp\" (UID: \"25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-b5dbp"
Oct 31 01:15:00.415267 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.415107    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2wfl\" (UniqueName: \"kubernetes.io/projected/0f3a0a27-f945-4566-9421-31d96a47f8f4-kube-api-access-m2wfl\") pod \"maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885\" (UID: \"0f3a0a27-f945-4566-9421-31d96a47f8f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885"
Oct 31 01:15:00.415399 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.415378    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qrhr\" (UniqueName: \"kubernetes.io/projected/25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3-kube-api-access-8qrhr\") pod \"maas-api-key-cleanup-29890155-b5dbp\" (UID: \"25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-b5dbp"
Oct 31 01:15:00.516694 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.516643    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpwwb\" (UniqueName: \"kubernetes.io/projected/b5fdc538-6826-42ad-a662-5ed5ed9fc004-kube-api-access-qpwwb\") pod \"maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p\" (UID: \"b5fdc538-6826-42ad-a662-5ed5ed9fc004\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p"
Oct 31 01:15:00.516910 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.516736    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5fdc538-6826-42ad-a662-5ed5ed9fc004-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p\" (UID: \"b5fdc538-6826-42ad-a662-5ed5ed9fc004\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p"
Oct 31 01:15:00.516910 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.516776    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-559jz\" (UniqueName: \"kubernetes.io/projected/79d8c2c4-a237-4c5b-9522-d64a581b918e-kube-api-access-559jz\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl\" (UID: \"79d8c2c4-a237-4c5b-9522-d64a581b918e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl"
Oct 31 01:15:00.516910 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.516848    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f3a0a27-f945-4566-9421-31d96a47f8f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885\" (UID: \"0f3a0a27-f945-4566-9421-31d96a47f8f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885"
Oct 31 01:15:00.516910 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.516882    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890155-b5dbp\" (UID: \"25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-b5dbp"
Oct 31 01:15:00.517136 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.516951    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79d8c2c4-a237-4c5b-9522-d64a581b918e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl\" (UID: \"79d8c2c4-a237-4c5b-9522-d64a581b918e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl"
Oct 31 01:15:00.517136 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.517072    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2wfl\" (UniqueName: \"kubernetes.io/projected/0f3a0a27-f945-4566-9421-31d96a47f8f4-kube-api-access-m2wfl\") pod \"maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885\" (UID: \"0f3a0a27-f945-4566-9421-31d96a47f8f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885"
Oct 31 01:15:00.517301 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.517281    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qrhr\" (UniqueName: \"kubernetes.io/projected/25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3-kube-api-access-8qrhr\") pod \"maas-api-key-cleanup-29890155-b5dbp\" (UID: \"25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-b5dbp"
Oct 31 01:15:00.517416 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.517391    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5fdc538-6826-42ad-a662-5ed5ed9fc004-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p\" (UID: \"b5fdc538-6826-42ad-a662-5ed5ed9fc004\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p"
Oct 31 01:15:00.517665 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.517644    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f3a0a27-f945-4566-9421-31d96a47f8f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885\" (UID: \"0f3a0a27-f945-4566-9421-31d96a47f8f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885"
Oct 31 01:15:00.517769 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.517651    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890155-b5dbp\" (UID: \"25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-b5dbp"
Oct 31 01:15:00.540898 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.540832    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2wfl\" (UniqueName: \"kubernetes.io/projected/0f3a0a27-f945-4566-9421-31d96a47f8f4-kube-api-access-m2wfl\") pod \"maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885\" (UID: \"0f3a0a27-f945-4566-9421-31d96a47f8f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885"
Oct 31 01:15:00.540898 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.540882    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qrhr\" (UniqueName: \"kubernetes.io/projected/25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3-kube-api-access-8qrhr\") pod \"maas-api-key-cleanup-29890155-b5dbp\" (UID: \"25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-b5dbp"
Oct 31 01:15:00.541063 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.540920    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpwwb\" (UniqueName: \"kubernetes.io/projected/b5fdc538-6826-42ad-a662-5ed5ed9fc004-kube-api-access-qpwwb\") pod \"maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p\" (UID: \"b5fdc538-6826-42ad-a662-5ed5ed9fc004\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p"
Oct 31 01:15:00.608154 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.608118    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-b5dbp"
Oct 31 01:15:00.618190 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.618163    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-559jz\" (UniqueName: \"kubernetes.io/projected/79d8c2c4-a237-4c5b-9522-d64a581b918e-kube-api-access-559jz\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl\" (UID: \"79d8c2c4-a237-4c5b-9522-d64a581b918e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl"
Oct 31 01:15:00.618277 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.618249    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79d8c2c4-a237-4c5b-9522-d64a581b918e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl\" (UID: \"79d8c2c4-a237-4c5b-9522-d64a581b918e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl"
Oct 31 01:15:00.618950 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.618925    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79d8c2c4-a237-4c5b-9522-d64a581b918e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl\" (UID: \"79d8c2c4-a237-4c5b-9522-d64a581b918e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl"
Oct 31 01:15:00.641816 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.641788    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-559jz\" (UniqueName: \"kubernetes.io/projected/79d8c2c4-a237-4c5b-9522-d64a581b918e-kube-api-access-559jz\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl\" (UID: \"79d8c2c4-a237-4c5b-9522-d64a581b918e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl"
Oct 31 01:15:00.649941 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.649911    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p"
Oct 31 01:15:00.683071 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.683035    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885"
Oct 31 01:15:00.694855 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.694749    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl"
Oct 31 01:15:00.935198 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:15:00.935150    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f9fe0d_fa0c_4b35_9cd3_33d2ded119b3.slice/crio-48ff4e2eb5d522f0a95c0dd9338438799375232ae32f274b7d0e57fc8b881f8b WatchSource:0}: Error finding container 48ff4e2eb5d522f0a95c0dd9338438799375232ae32f274b7d0e57fc8b881f8b: Status 404 returned error can't find the container with id 48ff4e2eb5d522f0a95c0dd9338438799375232ae32f274b7d0e57fc8b881f8b
Oct 31 01:15:00.936505 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.936479    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-b5dbp"]
Oct 31 01:15:00.978165 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:15:00.978139    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5fdc538_6826_42ad_a662_5ed5ed9fc004.slice/crio-25ec36034ee98c4586971c17eae7c61dd1980e42d9794f18bc6b2443a2f1e288 WatchSource:0}: Error finding container 25ec36034ee98c4586971c17eae7c61dd1980e42d9794f18bc6b2443a2f1e288: Status 404 returned error can't find the container with id 25ec36034ee98c4586971c17eae7c61dd1980e42d9794f18bc6b2443a2f1e288
Oct 31 01:15:00.980154 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:00.979631    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p"]
Oct 31 01:15:01.035717 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:01.035664    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885"]
Oct 31 01:15:01.041175 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:01.041148    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl"]
Oct 31 01:15:01.121468 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:15:01.121421    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f3a0a27_f945_4566_9421_31d96a47f8f4.slice/crio-be4ce11d1943dda930b9de3204681ed736a93e13fe56fb26e8c3110f5f64be3a WatchSource:0}: Error finding container be4ce11d1943dda930b9de3204681ed736a93e13fe56fb26e8c3110f5f64be3a: Status 404 returned error can't find the container with id be4ce11d1943dda930b9de3204681ed736a93e13fe56fb26e8c3110f5f64be3a
Oct 31 01:15:01.121800 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:15:01.121765    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d8c2c4_a237_4c5b_9522_d64a581b918e.slice/crio-298a8a7e96b5e4572bfc528634d85a52011a35372ef30d1dc60c80d9a933e6ec WatchSource:0}: Error finding container 298a8a7e96b5e4572bfc528634d85a52011a35372ef30d1dc60c80d9a933e6ec: Status 404 returned error can't find the container with id 298a8a7e96b5e4572bfc528634d85a52011a35372ef30d1dc60c80d9a933e6ec
Oct 31 01:15:01.847542 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:01.847450    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p" event={"ID":"b5fdc538-6826-42ad-a662-5ed5ed9fc004","Type":"ContainerStarted","Data":"f265219b09468de5707392395cabe52c146dcce6a30a0a86aa3c731312a2f6bc"}
Oct 31 01:15:01.847542 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:01.847521    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p" event={"ID":"b5fdc538-6826-42ad-a662-5ed5ed9fc004","Type":"ContainerStarted","Data":"25ec36034ee98c4586971c17eae7c61dd1980e42d9794f18bc6b2443a2f1e288"}
Oct 31 01:15:01.849318 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:01.849291    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl" event={"ID":"79d8c2c4-a237-4c5b-9522-d64a581b918e","Type":"ContainerStarted","Data":"5af71821d0696b528c51ed62b9791e2611ff4bb07e8e7178b959ae6fff6553e7"}
Oct 31 01:15:01.849442 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:01.849336    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl" event={"ID":"79d8c2c4-a237-4c5b-9522-d64a581b918e","Type":"ContainerStarted","Data":"298a8a7e96b5e4572bfc528634d85a52011a35372ef30d1dc60c80d9a933e6ec"}
Oct 31 01:15:01.850984 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:01.850958    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885" event={"ID":"0f3a0a27-f945-4566-9421-31d96a47f8f4","Type":"ContainerStarted","Data":"51ad61e973b74f4534670a57000aa61148dab3eb1971a0e4c5e1f4aab49f8310"}
Oct 31 01:15:01.851072 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:01.851007    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885" event={"ID":"0f3a0a27-f945-4566-9421-31d96a47f8f4","Type":"ContainerStarted","Data":"be4ce11d1943dda930b9de3204681ed736a93e13fe56fb26e8c3110f5f64be3a"}
Oct 31 01:15:01.852547 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:01.852519    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-b5dbp" event={"ID":"25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3","Type":"ContainerStarted","Data":"a876dfd05adbc71584575e539e211bab438668826474e2ecc2a94cae98fb1143"}
Oct 31 01:15:01.852655 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:01.852569    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-b5dbp" event={"ID":"25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3","Type":"ContainerStarted","Data":"48ff4e2eb5d522f0a95c0dd9338438799375232ae32f274b7d0e57fc8b881f8b"}
Oct 31 01:15:01.902533 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:01.902466    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p" podStartSLOduration=1.902448398 podStartE2EDuration="1.902448398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:01.901432204 +0000 UTC m=+3362.123467750" watchObservedRunningTime="2026-10-31 01:15:01.902448398 +0000 UTC m=+3362.124483943"
Oct 31 01:15:01.953873 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:01.953797    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl" podStartSLOduration=1.95377744 podStartE2EDuration="1.95377744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:01.945268105 +0000 UTC m=+3362.167303646" watchObservedRunningTime="2026-10-31 01:15:01.95377744 +0000 UTC m=+3362.175812984"
Oct 31 01:15:02.002161 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:02.002103    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-b5dbp" podStartSLOduration=2.002087439 podStartE2EDuration="2.002087439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:01.998144494 +0000 UTC m=+3362.220180026" watchObservedRunningTime="2026-10-31 01:15:02.002087439 +0000 UTC m=+3362.224122980"
Oct 31 01:15:02.054336 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:02.054283    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885" podStartSLOduration=2.054270677 podStartE2EDuration="2.054270677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:02.051410701 +0000 UTC m=+3362.273446244" watchObservedRunningTime="2026-10-31 01:15:02.054270677 +0000 UTC m=+3362.276306218"
Oct 31 01:15:02.862285 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:02.862249    4327 generic.go:356] "Generic (PLEG): container finished" podID="0f3a0a27-f945-4566-9421-31d96a47f8f4" containerID="51ad61e973b74f4534670a57000aa61148dab3eb1971a0e4c5e1f4aab49f8310" exitCode=60
Oct 31 01:15:02.862779 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:02.862331    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885" event={"ID":"0f3a0a27-f945-4566-9421-31d96a47f8f4","Type":"ContainerDied","Data":"51ad61e973b74f4534670a57000aa61148dab3eb1971a0e4c5e1f4aab49f8310"}
Oct 31 01:15:02.862779 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:02.862630    4327 scope.go:122] "RemoveContainer" containerID="51ad61e973b74f4534670a57000aa61148dab3eb1971a0e4c5e1f4aab49f8310"
Oct 31 01:15:02.864257 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:02.864231    4327 generic.go:356] "Generic (PLEG): container finished" podID="25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3" containerID="a876dfd05adbc71584575e539e211bab438668826474e2ecc2a94cae98fb1143" exitCode=60
Oct 31 01:15:02.864386 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:02.864317    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-b5dbp" event={"ID":"25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3","Type":"ContainerDied","Data":"a876dfd05adbc71584575e539e211bab438668826474e2ecc2a94cae98fb1143"}
Oct 31 01:15:02.864537 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:02.864523    4327 scope.go:122] "RemoveContainer" containerID="a876dfd05adbc71584575e539e211bab438668826474e2ecc2a94cae98fb1143"
Oct 31 01:15:02.866225 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:02.866127    4327 generic.go:356] "Generic (PLEG): container finished" podID="b5fdc538-6826-42ad-a662-5ed5ed9fc004" containerID="f265219b09468de5707392395cabe52c146dcce6a30a0a86aa3c731312a2f6bc" exitCode=60
Oct 31 01:15:02.866225 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:02.866201    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p" event={"ID":"b5fdc538-6826-42ad-a662-5ed5ed9fc004","Type":"ContainerDied","Data":"f265219b09468de5707392395cabe52c146dcce6a30a0a86aa3c731312a2f6bc"}
Oct 31 01:15:02.866503 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:02.866431    4327 scope.go:122] "RemoveContainer" containerID="f265219b09468de5707392395cabe52c146dcce6a30a0a86aa3c731312a2f6bc"
Oct 31 01:15:02.868259 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:02.868210    4327 generic.go:356] "Generic (PLEG): container finished" podID="79d8c2c4-a237-4c5b-9522-d64a581b918e" containerID="5af71821d0696b528c51ed62b9791e2611ff4bb07e8e7178b959ae6fff6553e7" exitCode=60
Oct 31 01:15:02.868259 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:02.868235    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl" event={"ID":"79d8c2c4-a237-4c5b-9522-d64a581b918e","Type":"ContainerDied","Data":"5af71821d0696b528c51ed62b9791e2611ff4bb07e8e7178b959ae6fff6553e7"}
Oct 31 01:15:02.868458 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:02.868444    4327 scope.go:122] "RemoveContainer" containerID="5af71821d0696b528c51ed62b9791e2611ff4bb07e8e7178b959ae6fff6553e7"
Oct 31 01:15:03.877587 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:03.877549    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl" event={"ID":"79d8c2c4-a237-4c5b-9522-d64a581b918e","Type":"ContainerStarted","Data":"3dfd36609ae48354ae1540aadc277421b18c5561258dee24c3777d8826d4aa98"}
Oct 31 01:15:03.879654 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:03.879624    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885" event={"ID":"0f3a0a27-f945-4566-9421-31d96a47f8f4","Type":"ContainerStarted","Data":"edc1544447dace6ca8ce573249e38482227d9500ea36bdc5a87148d26b74cf99"}
Oct 31 01:15:03.881489 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:03.881465    4327 generic.go:356] "Generic (PLEG): container finished" podID="25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3" containerID="9a0a99dfa649dcd18fcbcf5ee135f4d92519fff7ae9ffd0d135de8e5f0903659" exitCode=60
Oct 31 01:15:03.881606 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:03.881545    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-b5dbp" event={"ID":"25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3","Type":"ContainerDied","Data":"9a0a99dfa649dcd18fcbcf5ee135f4d92519fff7ae9ffd0d135de8e5f0903659"}
Oct 31 01:15:03.881606 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:03.881594    4327 scope.go:122] "RemoveContainer" containerID="a876dfd05adbc71584575e539e211bab438668826474e2ecc2a94cae98fb1143"
Oct 31 01:15:03.881796 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:03.881780    4327 scope.go:122] "RemoveContainer" containerID="9a0a99dfa649dcd18fcbcf5ee135f4d92519fff7ae9ffd0d135de8e5f0903659"
Oct 31 01:15:03.882098 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:15:03.882072    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890155-b5dbp_odh-ai-gateway-infra(25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-b5dbp" podUID="25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3"
Oct 31 01:15:03.883730 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:03.883708    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p" event={"ID":"b5fdc538-6826-42ad-a662-5ed5ed9fc004","Type":"ContainerStarted","Data":"e37a4e18811dfcb0af50dbff54d544b5179825578564ed76b4d0d37bfd76abc8"}
Oct 31 01:15:04.063483 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.063449    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p"]
Oct 31 01:15:04.181538 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.181509    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc"]
Oct 31 01:15:04.181746 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.181720    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc" podUID="98668443-03d5-49cf-9c91-29afc4bf3447" containerName="payload-pre-processing" containerID="cri-o://2b89019eae061948c94a5e2d6ed625c62cf3be8c807bcf64743a5c2199f25f2f" gracePeriod=30
Oct 31 01:15:04.562011 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.561315    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc"
Oct 31 01:15:04.703327 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.703295    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98668443-03d5-49cf-9c91-29afc4bf3447-kube-api-access-brn78\" (UniqueName: \"kubernetes.io/projected/98668443-03d5-49cf-9c91-29afc4bf3447-kube-api-access-brn78\") pod \"98668443-03d5-49cf-9c91-29afc4bf3447\" (UID: \"98668443-03d5-49cf-9c91-29afc4bf3447\") "
Oct 31 01:15:04.703501 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.703338    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98668443-03d5-49cf-9c91-29afc4bf3447-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98668443-03d5-49cf-9c91-29afc4bf3447-plugins-config-volume\") pod \"98668443-03d5-49cf-9c91-29afc4bf3447\" (UID: \"98668443-03d5-49cf-9c91-29afc4bf3447\") "
Oct 31 01:15:04.703788 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.703767    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98668443-03d5-49cf-9c91-29afc4bf3447-plugins-config-volume" pod "98668443-03d5-49cf-9c91-29afc4bf3447" (UID: "98668443-03d5-49cf-9c91-29afc4bf3447"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:04.703940 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.703921    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98668443-03d5-49cf-9c91-29afc4bf3447-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:15:04.705547 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.705524    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98668443-03d5-49cf-9c91-29afc4bf3447-kube-api-access-brn78" pod "98668443-03d5-49cf-9c91-29afc4bf3447" (UID: "98668443-03d5-49cf-9c91-29afc4bf3447"). InnerVolumeSpecName "kube-api-access-brn78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:04.805167 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.805083    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brn78\" (UniqueName: \"kubernetes.io/projected/98668443-03d5-49cf-9c91-29afc4bf3447-kube-api-access-brn78\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:15:04.894368 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.894336    4327 generic.go:356] "Generic (PLEG): container finished" podID="79d8c2c4-a237-4c5b-9522-d64a581b918e" containerID="3dfd36609ae48354ae1540aadc277421b18c5561258dee24c3777d8826d4aa98" exitCode=60
Oct 31 01:15:04.894862 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.894421    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl" event={"ID":"79d8c2c4-a237-4c5b-9522-d64a581b918e","Type":"ContainerDied","Data":"3dfd36609ae48354ae1540aadc277421b18c5561258dee24c3777d8826d4aa98"}
Oct 31 01:15:04.894862 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.894488    4327 scope.go:122] "RemoveContainer" containerID="5af71821d0696b528c51ed62b9791e2611ff4bb07e8e7178b959ae6fff6553e7"
Oct 31 01:15:04.894862 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.894808    4327 scope.go:122] "RemoveContainer" containerID="3dfd36609ae48354ae1540aadc277421b18c5561258dee24c3777d8826d4aa98"
Oct 31 01:15:04.895225 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:15:04.895197    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl_odh-ai-gateway-infra(79d8c2c4-a237-4c5b-9522-d64a581b918e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl" podUID="79d8c2c4-a237-4c5b-9522-d64a581b918e"
Oct 31 01:15:04.896621 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.896598    4327 generic.go:356] "Generic (PLEG): container finished" podID="0f3a0a27-f945-4566-9421-31d96a47f8f4" containerID="edc1544447dace6ca8ce573249e38482227d9500ea36bdc5a87148d26b74cf99" exitCode=60
Oct 31 01:15:04.896755 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.896691    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885" event={"ID":"0f3a0a27-f945-4566-9421-31d96a47f8f4","Type":"ContainerDied","Data":"edc1544447dace6ca8ce573249e38482227d9500ea36bdc5a87148d26b74cf99"}
Oct 31 01:15:04.897053 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.897032    4327 scope.go:122] "RemoveContainer" containerID="edc1544447dace6ca8ce573249e38482227d9500ea36bdc5a87148d26b74cf99"
Oct 31 01:15:04.898781 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:15:04.898748    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885_odh-ai-gateway-infra(0f3a0a27-f945-4566-9421-31d96a47f8f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885" podUID="0f3a0a27-f945-4566-9421-31d96a47f8f4"
Oct 31 01:15:04.899968 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.899943    4327 generic.go:356] "Generic (PLEG): container finished" podID="98668443-03d5-49cf-9c91-29afc4bf3447" containerID="2b89019eae061948c94a5e2d6ed625c62cf3be8c807bcf64743a5c2199f25f2f" exitCode=0
Oct 31 01:15:04.900466 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.900163    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc" event={"ID":"98668443-03d5-49cf-9c91-29afc4bf3447","Type":"ContainerDied","Data":"2b89019eae061948c94a5e2d6ed625c62cf3be8c807bcf64743a5c2199f25f2f"}
Oct 31 01:15:04.900466 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.900220    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc"
Oct 31 01:15:04.900466 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.900263    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc" event={"ID":"98668443-03d5-49cf-9c91-29afc4bf3447","Type":"ContainerDied","Data":"8afe741762c8438fd4971d557b1521c07751f0bc6e8c9ac88ac5e643e88bf37f"}
Oct 31 01:15:04.929148 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.909418    4327 scope.go:122] "RemoveContainer" containerID="9a0a99dfa649dcd18fcbcf5ee135f4d92519fff7ae9ffd0d135de8e5f0903659"
Oct 31 01:15:04.929148 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:15:04.913402    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890155-b5dbp_odh-ai-gateway-infra(25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-b5dbp" podUID="25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3"
Oct 31 01:15:04.929148 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.925350    4327 generic.go:356] "Generic (PLEG): container finished" podID="b5fdc538-6826-42ad-a662-5ed5ed9fc004" containerID="e37a4e18811dfcb0af50dbff54d544b5179825578564ed76b4d0d37bfd76abc8" exitCode=60
Oct 31 01:15:04.929148 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.925481    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p" event={"ID":"b5fdc538-6826-42ad-a662-5ed5ed9fc004","Type":"ContainerDied","Data":"e37a4e18811dfcb0af50dbff54d544b5179825578564ed76b4d0d37bfd76abc8"}
Oct 31 01:15:04.929148 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.925955    4327 scope.go:122] "RemoveContainer" containerID="e37a4e18811dfcb0af50dbff54d544b5179825578564ed76b4d0d37bfd76abc8"
Oct 31 01:15:04.929148 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:15:04.926548    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p_odh-ai-gateway-infra(b5fdc538-6826-42ad-a662-5ed5ed9fc004)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p" podUID="b5fdc538-6826-42ad-a662-5ed5ed9fc004"
Oct 31 01:15:04.940168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.940143    4327 scope.go:122] "RemoveContainer" containerID="51ad61e973b74f4534670a57000aa61148dab3eb1971a0e4c5e1f4aab49f8310"
Oct 31 01:15:04.954138 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.954118    4327 scope.go:122] "RemoveContainer" containerID="2b89019eae061948c94a5e2d6ed625c62cf3be8c807bcf64743a5c2199f25f2f"
Oct 31 01:15:04.965863 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.965843    4327 scope.go:122] "RemoveContainer" containerID="2b89019eae061948c94a5e2d6ed625c62cf3be8c807bcf64743a5c2199f25f2f"
Oct 31 01:15:04.966172 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:15:04.966146    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b89019eae061948c94a5e2d6ed625c62cf3be8c807bcf64743a5c2199f25f2f\": container with ID starting with 2b89019eae061948c94a5e2d6ed625c62cf3be8c807bcf64743a5c2199f25f2f not found: ID does not exist" containerID="2b89019eae061948c94a5e2d6ed625c62cf3be8c807bcf64743a5c2199f25f2f"
Oct 31 01:15:04.966277 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.966179    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b89019eae061948c94a5e2d6ed625c62cf3be8c807bcf64743a5c2199f25f2f"} err="failed to get container status \"2b89019eae061948c94a5e2d6ed625c62cf3be8c807bcf64743a5c2199f25f2f\": rpc error: code = NotFound desc = could not find container \"2b89019eae061948c94a5e2d6ed625c62cf3be8c807bcf64743a5c2199f25f2f\": container with ID starting with 2b89019eae061948c94a5e2d6ed625c62cf3be8c807bcf64743a5c2199f25f2f not found: ID does not exist"
Oct 31 01:15:04.966277 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:04.966199    4327 scope.go:122] "RemoveContainer" containerID="f265219b09468de5707392395cabe52c146dcce6a30a0a86aa3c731312a2f6bc"
Oct 31 01:15:05.131131 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:05.131098    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc"]
Oct 31 01:15:05.137567 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:05.137536    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75ead8-7c6d545b67-hggkc"]
Oct 31 01:15:05.923582 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:05.923548    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98668443-03d5-49cf-9c91-29afc4bf3447" path="/var/lib/kubelet/pods/98668443-03d5-49cf-9c91-29afc4bf3447/volumes"
Oct 31 01:15:05.934539 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:05.934513    4327 scope.go:122] "RemoveContainer" containerID="3dfd36609ae48354ae1540aadc277421b18c5561258dee24c3777d8826d4aa98"
Oct 31 01:15:05.934917 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:15:05.934887    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl_odh-ai-gateway-infra(79d8c2c4-a237-4c5b-9522-d64a581b918e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl" podUID="79d8c2c4-a237-4c5b-9522-d64a581b918e"
Oct 31 01:15:05.936235 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:05.936217    4327 scope.go:122] "RemoveContainer" containerID="edc1544447dace6ca8ce573249e38482227d9500ea36bdc5a87148d26b74cf99"
Oct 31 01:15:05.936527 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:15:05.936508    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885_odh-ai-gateway-infra(0f3a0a27-f945-4566-9421-31d96a47f8f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885" podUID="0f3a0a27-f945-4566-9421-31d96a47f8f4"
Oct 31 01:15:06.112124 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:06.112096    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p"
Oct 31 01:15:06.234127 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:06.234047    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5fdc538-6826-42ad-a662-5ed5ed9fc004-kube-api-access-qpwwb\" (UniqueName: \"kubernetes.io/projected/b5fdc538-6826-42ad-a662-5ed5ed9fc004-kube-api-access-qpwwb\") pod \"b5fdc538-6826-42ad-a662-5ed5ed9fc004\" (UID: \"b5fdc538-6826-42ad-a662-5ed5ed9fc004\") "
Oct 31 01:15:06.234283 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:06.234212    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5fdc538-6826-42ad-a662-5ed5ed9fc004-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5fdc538-6826-42ad-a662-5ed5ed9fc004-maas-api-service-ca\") pod \"b5fdc538-6826-42ad-a662-5ed5ed9fc004\" (UID: \"b5fdc538-6826-42ad-a662-5ed5ed9fc004\") "
Oct 31 01:15:06.234631 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:06.234587    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5fdc538-6826-42ad-a662-5ed5ed9fc004-maas-api-service-ca" pod "b5fdc538-6826-42ad-a662-5ed5ed9fc004" (UID: "b5fdc538-6826-42ad-a662-5ed5ed9fc004"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:06.234933 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:06.234904    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5fdc538-6826-42ad-a662-5ed5ed9fc004-maas-api-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:15:06.236350 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:06.236322    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5fdc538-6826-42ad-a662-5ed5ed9fc004-kube-api-access-qpwwb" pod "b5fdc538-6826-42ad-a662-5ed5ed9fc004" (UID: "b5fdc538-6826-42ad-a662-5ed5ed9fc004"). InnerVolumeSpecName "kube-api-access-qpwwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:06.336297 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:06.336263    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpwwb\" (UniqueName: \"kubernetes.io/projected/b5fdc538-6826-42ad-a662-5ed5ed9fc004-kube-api-access-qpwwb\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:15:06.948063 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:06.948029    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p" event={"ID":"b5fdc538-6826-42ad-a662-5ed5ed9fc004","Type":"ContainerDied","Data":"25ec36034ee98c4586971c17eae7c61dd1980e42d9794f18bc6b2443a2f1e288"}
Oct 31 01:15:06.948561 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:06.948093    4327 scope.go:122] "RemoveContainer" containerID="e37a4e18811dfcb0af50dbff54d544b5179825578564ed76b4d0d37bfd76abc8"
Oct 31 01:15:06.948561 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:06.948042    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p"
Oct 31 01:15:07.002256 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:07.002228    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p"]
Oct 31 01:15:07.014595 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:07.014567    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-75ead8-29890155-4jn7p"]
Oct 31 01:15:07.923541 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:07.923507    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5fdc538-6826-42ad-a662-5ed5ed9fc004" path="/var/lib/kubelet/pods/b5fdc538-6826-42ad-a662-5ed5ed9fc004/volumes"
Oct 31 01:15:10.452587 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:10.452550    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2"]
Oct 31 01:15:10.453324 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:10.453309    4327 cpu_manager.go:414] "containerMap: removing container" podUID="b5fdc538-6826-42ad-a662-5ed5ed9fc004" containerName="cleanup"
Oct 31 01:15:10.453324 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:10.453324    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5fdc538-6826-42ad-a662-5ed5ed9fc004" containerName="cleanup"
Oct 31 01:15:10.453418 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:10.453348    4327 cpu_manager.go:414] "containerMap: removing container" podUID="b5fdc538-6826-42ad-a662-5ed5ed9fc004" containerName="cleanup"
Oct 31 01:15:10.453418 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:10.453354    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5fdc538-6826-42ad-a662-5ed5ed9fc004" containerName="cleanup"
Oct 31 01:15:10.453418 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:10.453379    4327 cpu_manager.go:414] "containerMap: removing container" podUID="98668443-03d5-49cf-9c91-29afc4bf3447" containerName="payload-pre-processing"
Oct 31 01:15:10.453418 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:10.453386    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98668443-03d5-49cf-9c91-29afc4bf3447" containerName="payload-pre-processing"
Oct 31 01:15:10.453542 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:10.453483    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5fdc538-6826-42ad-a662-5ed5ed9fc004" containerName="cleanup"
Oct 31 01:15:10.453542 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:10.453495    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="98668443-03d5-49cf-9c91-29afc4bf3447" containerName="payload-pre-processing"
Oct 31 01:15:10.453542 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:10.453506    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5fdc538-6826-42ad-a662-5ed5ed9fc004" containerName="cleanup"
Oct 31 01:15:10.458357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:10.458339    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2"
Oct 31 01:15:10.478401 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:10.478372    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-fa97ea-serving-cert\""
Oct 31 01:15:10.480365 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:10.480336    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2"]
Oct 31 01:15:10.597805 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:10.597775    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55c787a6-2c98-4474-ac32-82f652780c14-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2\" (UID: \"55c787a6-2c98-4474-ac32-82f652780c14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2"
Oct 31 01:15:10.597989 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:10.597846    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd579\" (UniqueName: \"kubernetes.io/projected/55c787a6-2c98-4474-ac32-82f652780c14-kube-api-access-rd579\") pod \"maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2\" (UID: \"55c787a6-2c98-4474-ac32-82f652780c14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2"
Oct 31 01:15:10.598030 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:10.597989    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55c787a6-2c98-4474-ac32-82f652780c14-metrics-certs\") pod \"maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2\" (UID: \"55c787a6-2c98-4474-ac32-82f652780c14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2"
Oct 31 01:15:10.598066 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:10.598039    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c787a6-2c98-4474-ac32-82f652780c14-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2\" (UID: \"55c787a6-2c98-4474-ac32-82f652780c14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2"
Oct 31 01:15:10.703754 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:10.702949    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55c787a6-2c98-4474-ac32-82f652780c14-metrics-certs\") pod \"maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2\" (UID: \"55c787a6-2c98-4474-ac32-82f652780c14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2"
Oct 31 01:15:10.703754 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:10.703033    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c787a6-2c98-4474-ac32-82f652780c14-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2\" (UID: \"55c787a6-2c98-4474-ac32-82f652780c14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2"
Oct 31 01:15:10.703754 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:10.703193    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55c787a6-2c98-4474-ac32-82f652780c14-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2\" (UID: \"55c787a6-2c98-4474-ac32-82f652780c14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2"
Oct 31 01:15:10.703754 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:10.703296    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd579\" (UniqueName: \"kubernetes.io/projected/55c787a6-2c98-4474-ac32-82f652780c14-kube-api-access-rd579\") pod \"maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2\" (UID: \"55c787a6-2c98-4474-ac32-82f652780c14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2"
Oct 31 01:15:10.705267 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:10.705236    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c787a6-2c98-4474-ac32-82f652780c14-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2\" (UID: \"55c787a6-2c98-4474-ac32-82f652780c14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2"
Oct 31 01:15:10.725823 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:10.725787    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55c787a6-2c98-4474-ac32-82f652780c14-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2\" (UID: \"55c787a6-2c98-4474-ac32-82f652780c14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2"
Oct 31 01:15:10.745939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:10.745902    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd579\" (UniqueName: \"kubernetes.io/projected/55c787a6-2c98-4474-ac32-82f652780c14-kube-api-access-rd579\") pod \"maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2\" (UID: \"55c787a6-2c98-4474-ac32-82f652780c14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2"
Oct 31 01:15:10.755112 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:10.755080    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55c787a6-2c98-4474-ac32-82f652780c14-metrics-certs\") pod \"maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2\" (UID: \"55c787a6-2c98-4474-ac32-82f652780c14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2"
Oct 31 01:15:10.773242 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:10.773205    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2"
Oct 31 01:15:11.103100 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:15:11.103070    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c787a6_2c98_4474_ac32_82f652780c14.slice/crio-ddd9239282420697311dad97d97e7b86033c65f1cf3c3e6cc58a1ff8c49e41fd WatchSource:0}: Error finding container ddd9239282420697311dad97d97e7b86033c65f1cf3c3e6cc58a1ff8c49e41fd: Status 404 returned error can't find the container with id ddd9239282420697311dad97d97e7b86033c65f1cf3c3e6cc58a1ff8c49e41fd
Oct 31 01:15:11.104360 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:11.104336    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2"]
Oct 31 01:15:11.997754 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:11.997715    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2" event={"ID":"55c787a6-2c98-4474-ac32-82f652780c14","Type":"ContainerStarted","Data":"36c6257e79f22ebfd38d8b7858890d9ade7839f3b77eea8e5ca78585bf9cd9aa"}
Oct 31 01:15:11.998189 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:11.997787    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2" event={"ID":"55c787a6-2c98-4474-ac32-82f652780c14","Type":"ContainerStarted","Data":"ddd9239282420697311dad97d97e7b86033c65f1cf3c3e6cc58a1ff8c49e41fd"}
Oct 31 01:15:11.998189 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:11.997818    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2"
Oct 31 01:15:12.078460 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:12.078401    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2" podStartSLOduration=1.705104972 podStartE2EDuration="2.078385323s" totalImagesPullingTime="373.280351ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:12.065490318 +0000 UTC m=+3372.287525860" watchObservedRunningTime="2026-10-31 01:15:12.078385323 +0000 UTC m=+3372.300420917"
Oct 31 01:15:15.294595 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:15:15.294554    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:15:16.913615 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:16.913585    4327 scope.go:122] "RemoveContainer" containerID="9a0a99dfa649dcd18fcbcf5ee135f4d92519fff7ae9ffd0d135de8e5f0903659"
Oct 31 01:15:17.913787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:17.913753    4327 scope.go:122] "RemoveContainer" containerID="3dfd36609ae48354ae1540aadc277421b18c5561258dee24c3777d8826d4aa98"
Oct 31 01:15:18.010208 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:18.010179    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2"
Oct 31 01:15:18.048834 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:18.048802    4327 generic.go:356] "Generic (PLEG): container finished" podID="25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3" containerID="a8d2845fa882b505a1e1434e34b8d9b348c0290b5afdf8ffbca6256a3bcf089b" exitCode=60
Oct 31 01:15:18.049007 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:18.048880    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-b5dbp" event={"ID":"25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3","Type":"ContainerDied","Data":"a8d2845fa882b505a1e1434e34b8d9b348c0290b5afdf8ffbca6256a3bcf089b"}
Oct 31 01:15:18.049007 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:18.048945    4327 scope.go:122] "RemoveContainer" containerID="9a0a99dfa649dcd18fcbcf5ee135f4d92519fff7ae9ffd0d135de8e5f0903659"
Oct 31 01:15:18.049466 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:18.049437    4327 scope.go:122] "RemoveContainer" containerID="a8d2845fa882b505a1e1434e34b8d9b348c0290b5afdf8ffbca6256a3bcf089b"
Oct 31 01:15:18.049889 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:15:18.049864    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890155-b5dbp_odh-ai-gateway-infra(25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-b5dbp" podUID="25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3"
Oct 31 01:15:18.595155 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:18.595075    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"]
Oct 31 01:15:18.627637 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:18.627603    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"]
Oct 31 01:15:18.914298 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:18.914272    4327 scope.go:122] "RemoveContainer" containerID="edc1544447dace6ca8ce573249e38482227d9500ea36bdc5a87148d26b74cf99"
Oct 31 01:15:19.059883 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:19.059850    4327 generic.go:356] "Generic (PLEG): container finished" podID="79d8c2c4-a237-4c5b-9522-d64a581b918e" containerID="90e4052f1692021485f2df1cc4ee9ab5341908e6b9666ec63858b3d4d5d89d05" exitCode=60
Oct 31 01:15:19.060089 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:19.059924    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl" event={"ID":"79d8c2c4-a237-4c5b-9522-d64a581b918e","Type":"ContainerDied","Data":"90e4052f1692021485f2df1cc4ee9ab5341908e6b9666ec63858b3d4d5d89d05"}
Oct 31 01:15:19.060089 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:19.059989    4327 scope.go:122] "RemoveContainer" containerID="3dfd36609ae48354ae1540aadc277421b18c5561258dee24c3777d8826d4aa98"
Oct 31 01:15:19.060478 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:19.060452    4327 scope.go:122] "RemoveContainer" containerID="90e4052f1692021485f2df1cc4ee9ab5341908e6b9666ec63858b3d4d5d89d05"
Oct 31 01:15:19.060912 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:15:19.060887    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl_odh-ai-gateway-infra(79d8c2c4-a237-4c5b-9522-d64a581b918e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl" podUID="79d8c2c4-a237-4c5b-9522-d64a581b918e"
Oct 31 01:15:19.139755 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:19.139722    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-b5dbp"]
Oct 31 01:15:19.358016 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:19.357984    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-b5dbp"
Oct 31 01:15:19.438906 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:19.438873    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3-maas-api-service-ca\") pod \"25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3\" (UID: \"25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3\") "
Oct 31 01:15:19.439070 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:19.439047    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3-kube-api-access-8qrhr\" (UniqueName: \"kubernetes.io/projected/25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3-kube-api-access-8qrhr\") pod \"25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3\" (UID: \"25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3\") "
Oct 31 01:15:19.444130 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:19.444098    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3-maas-api-service-ca" pod "25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3" (UID: "25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:19.445483 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:19.445459    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3-maas-api-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:15:19.451454 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:19.451423    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3-kube-api-access-8qrhr" pod "25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3" (UID: "25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3"). InnerVolumeSpecName "kube-api-access-8qrhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:19.547079 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:19.547042    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qrhr\" (UniqueName: \"kubernetes.io/projected/25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3-kube-api-access-8qrhr\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:15:20.158007 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:20.157951    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl"]
Oct 31 01:15:20.163333 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:20.163302    4327 generic.go:356] "Generic (PLEG): container finished" podID="0f3a0a27-f945-4566-9421-31d96a47f8f4" containerID="0102a815034a1d5bf8598567f10fba9b428d0786f9a57ec3abcc6cd1ad1959c8" exitCode=60
Oct 31 01:15:20.163556 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:20.163366    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885" event={"ID":"0f3a0a27-f945-4566-9421-31d96a47f8f4","Type":"ContainerDied","Data":"0102a815034a1d5bf8598567f10fba9b428d0786f9a57ec3abcc6cd1ad1959c8"}
Oct 31 01:15:20.163651 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:20.163607    4327 scope.go:122] "RemoveContainer" containerID="edc1544447dace6ca8ce573249e38482227d9500ea36bdc5a87148d26b74cf99"
Oct 31 01:15:20.164116 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:20.164091    4327 scope.go:122] "RemoveContainer" containerID="0102a815034a1d5bf8598567f10fba9b428d0786f9a57ec3abcc6cd1ad1959c8"
Oct 31 01:15:20.164447 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:15:20.164420    4327 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885_odh-ai-gateway-infra(0f3a0a27-f945-4566-9421-31d96a47f8f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885" podUID="0f3a0a27-f945-4566-9421-31d96a47f8f4"
Oct 31 01:15:20.168256 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:20.168158    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-b5dbp" event={"ID":"25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3","Type":"ContainerDied","Data":"48ff4e2eb5d522f0a95c0dd9338438799375232ae32f274b7d0e57fc8b881f8b"}
Oct 31 01:15:20.170911 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:20.168468    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-b5dbp"
Oct 31 01:15:20.189625 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:20.189547    4327 scope.go:122] "RemoveContainer" containerID="a8d2845fa882b505a1e1434e34b8d9b348c0290b5afdf8ffbca6256a3bcf089b"
Oct 31 01:15:20.296337 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:20.296300    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-b5dbp"]
Oct 31 01:15:20.303864 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:20.303830    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-b5dbp"]
Oct 31 01:15:20.390710 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:20.387302    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl"
Oct 31 01:15:20.467989 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:20.467890    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d8c2c4-a237-4c5b-9522-d64a581b918e-kube-api-access-559jz\" (UniqueName: \"kubernetes.io/projected/79d8c2c4-a237-4c5b-9522-d64a581b918e-kube-api-access-559jz\") pod \"79d8c2c4-a237-4c5b-9522-d64a581b918e\" (UID: \"79d8c2c4-a237-4c5b-9522-d64a581b918e\") "
Oct 31 01:15:20.468226 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:20.468130    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79d8c2c4-a237-4c5b-9522-d64a581b918e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79d8c2c4-a237-4c5b-9522-d64a581b918e-maas-api-service-ca\") pod \"79d8c2c4-a237-4c5b-9522-d64a581b918e\" (UID: \"79d8c2c4-a237-4c5b-9522-d64a581b918e\") "
Oct 31 01:15:20.470263 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:20.470215    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d8c2c4-a237-4c5b-9522-d64a581b918e-maas-api-service-ca" pod "79d8c2c4-a237-4c5b-9522-d64a581b918e" (UID: "79d8c2c4-a237-4c5b-9522-d64a581b918e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:20.488039 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:20.487999    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d8c2c4-a237-4c5b-9522-d64a581b918e-kube-api-access-559jz" pod "79d8c2c4-a237-4c5b-9522-d64a581b918e" (UID: "79d8c2c4-a237-4c5b-9522-d64a581b918e"). InnerVolumeSpecName "kube-api-access-559jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:20.571146 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:20.571101    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-559jz\" (UniqueName: \"kubernetes.io/projected/79d8c2c4-a237-4c5b-9522-d64a581b918e-kube-api-access-559jz\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:15:20.571146 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:20.571135    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79d8c2c4-a237-4c5b-9522-d64a581b918e-maas-api-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:15:21.178182 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:21.178155    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl"
Oct 31 01:15:21.178646 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:21.178154    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl" event={"ID":"79d8c2c4-a237-4c5b-9522-d64a581b918e","Type":"ContainerDied","Data":"298a8a7e96b5e4572bfc528634d85a52011a35372ef30d1dc60c80d9a933e6ec"}
Oct 31 01:15:21.178646 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:21.178296    4327 scope.go:122] "RemoveContainer" containerID="90e4052f1692021485f2df1cc4ee9ab5341908e6b9666ec63858b3d4d5d89d05"
Oct 31 01:15:21.234712 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:21.234642    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl"]
Oct 31 01:15:21.238757 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:21.238731    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890155-kmgkl"]
Oct 31 01:15:21.267267 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:21.267238    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885"]
Oct 31 01:15:21.441487 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:21.441402    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885"
Oct 31 01:15:21.484705 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:21.484660    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f3a0a27-f945-4566-9421-31d96a47f8f4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f3a0a27-f945-4566-9421-31d96a47f8f4-maas-api-service-ca\") pod \"0f3a0a27-f945-4566-9421-31d96a47f8f4\" (UID: \"0f3a0a27-f945-4566-9421-31d96a47f8f4\") "
Oct 31 01:15:21.487245 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:21.484763    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f3a0a27-f945-4566-9421-31d96a47f8f4-kube-api-access-m2wfl\" (UniqueName: \"kubernetes.io/projected/0f3a0a27-f945-4566-9421-31d96a47f8f4-kube-api-access-m2wfl\") pod \"0f3a0a27-f945-4566-9421-31d96a47f8f4\" (UID: \"0f3a0a27-f945-4566-9421-31d96a47f8f4\") "
Oct 31 01:15:21.487245 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:21.486091    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f3a0a27-f945-4566-9421-31d96a47f8f4-maas-api-service-ca" pod "0f3a0a27-f945-4566-9421-31d96a47f8f4" (UID: "0f3a0a27-f945-4566-9421-31d96a47f8f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:21.488378 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:21.488347    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f3a0a27-f945-4566-9421-31d96a47f8f4-kube-api-access-m2wfl" pod "0f3a0a27-f945-4566-9421-31d96a47f8f4" (UID: "0f3a0a27-f945-4566-9421-31d96a47f8f4"). InnerVolumeSpecName "kube-api-access-m2wfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:21.587160 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:21.587121    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2wfl\" (UniqueName: \"kubernetes.io/projected/0f3a0a27-f945-4566-9421-31d96a47f8f4-kube-api-access-m2wfl\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:15:21.587160 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:21.587160    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f3a0a27-f945-4566-9421-31d96a47f8f4-maas-api-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:15:21.810516 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:21.810429    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw"]
Oct 31 01:15:21.853787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:21.853756    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j"]
Oct 31 01:15:21.942536 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:21.936100    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3" path="/var/lib/kubelet/pods/25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3/volumes"
Oct 31 01:15:21.942536 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:21.936632    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79d8c2c4-a237-4c5b-9522-d64a581b918e" path="/var/lib/kubelet/pods/79d8c2c4-a237-4c5b-9522-d64a581b918e/volumes"
Oct 31 01:15:22.192384 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:22.192337    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885" event={"ID":"0f3a0a27-f945-4566-9421-31d96a47f8f4","Type":"ContainerDied","Data":"be4ce11d1943dda930b9de3204681ed736a93e13fe56fb26e8c3110f5f64be3a"}
Oct 31 01:15:22.192871 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:22.192447    4327 scope.go:122] "RemoveContainer" containerID="0102a815034a1d5bf8598567f10fba9b428d0786f9a57ec3abcc6cd1ad1959c8"
Oct 31 01:15:22.193086 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:22.193067    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885"
Oct 31 01:15:22.278732 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:22.278212    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885"]
Oct 31 01:15:22.284002 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:22.283977    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-33d44f-29890155-n9885"]
Oct 31 01:15:23.981097 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:23.981055    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f3a0a27-f945-4566-9421-31d96a47f8f4" path="/var/lib/kubelet/pods/0f3a0a27-f945-4566-9421-31d96a47f8f4/volumes"
Oct 31 01:15:26.168751 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.164356    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b"]
Oct 31 01:15:26.190143 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.190106    4327 cpu_manager.go:414] "containerMap: removing container" podUID="79d8c2c4-a237-4c5b-9522-d64a581b918e" containerName="cleanup"
Oct 31 01:15:26.190360 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.190343    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d8c2c4-a237-4c5b-9522-d64a581b918e" containerName="cleanup"
Oct 31 01:15:26.190482 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.190471    4327 cpu_manager.go:414] "containerMap: removing container" podUID="0f3a0a27-f945-4566-9421-31d96a47f8f4" containerName="cleanup"
Oct 31 01:15:26.190564 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.190554    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f3a0a27-f945-4566-9421-31d96a47f8f4" containerName="cleanup"
Oct 31 01:15:26.190702 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.190671    4327 cpu_manager.go:414] "containerMap: removing container" podUID="0f3a0a27-f945-4566-9421-31d96a47f8f4" containerName="cleanup"
Oct 31 01:15:26.190810 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.190797    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f3a0a27-f945-4566-9421-31d96a47f8f4" containerName="cleanup"
Oct 31 01:15:26.190926 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.190915    4327 cpu_manager.go:414] "containerMap: removing container" podUID="25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3" containerName="cleanup"
Oct 31 01:15:26.191007 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.190997    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3" containerName="cleanup"
Oct 31 01:15:26.191109 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.191099    4327 cpu_manager.go:414] "containerMap: removing container" podUID="79d8c2c4-a237-4c5b-9522-d64a581b918e" containerName="cleanup"
Oct 31 01:15:26.191187 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.191177    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d8c2c4-a237-4c5b-9522-d64a581b918e" containerName="cleanup"
Oct 31 01:15:26.191269 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.191259    4327 cpu_manager.go:414] "containerMap: removing container" podUID="25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3" containerName="cleanup"
Oct 31 01:15:26.191341 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.191331    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3" containerName="cleanup"
Oct 31 01:15:26.191443 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.191426    4327 cpu_manager.go:414] "containerMap: removing container" podUID="25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3" containerName="cleanup"
Oct 31 01:15:26.191534 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.191523    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3" containerName="cleanup"
Oct 31 01:15:26.191620 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.191610    4327 cpu_manager.go:414] "containerMap: removing container" podUID="0f3a0a27-f945-4566-9421-31d96a47f8f4" containerName="cleanup"
Oct 31 01:15:26.191714 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.191704    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f3a0a27-f945-4566-9421-31d96a47f8f4" containerName="cleanup"
Oct 31 01:15:26.192051 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.192037    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3" containerName="cleanup"
Oct 31 01:15:26.192147 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.192138    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3" containerName="cleanup"
Oct 31 01:15:26.192231 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.192224    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f3a0a27-f945-4566-9421-31d96a47f8f4" containerName="cleanup"
Oct 31 01:15:26.192302 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.192295    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d8c2c4-a237-4c5b-9522-d64a581b918e" containerName="cleanup"
Oct 31 01:15:26.192380 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.192373    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f3a0a27-f945-4566-9421-31d96a47f8f4" containerName="cleanup"
Oct 31 01:15:26.192453 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.192446    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d8c2c4-a237-4c5b-9522-d64a581b918e" containerName="cleanup"
Oct 31 01:15:26.238990 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.238949    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b"]
Oct 31 01:15:26.257749 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.255930    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b"
Oct 31 01:15:26.273123 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.273096    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4207cd\""
Oct 31 01:15:26.285554 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.285530    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4207cd-dockercfg-dx9rx\""
Oct 31 01:15:26.358387 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.358342    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54718bb1-f732-4c6b-aa42-7eda6ecec318-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b\" (UID: \"54718bb1-f732-4c6b-aa42-7eda6ecec318\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b"
Oct 31 01:15:26.358600 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.358511    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgrkr\" (UniqueName: \"kubernetes.io/projected/54718bb1-f732-4c6b-aa42-7eda6ecec318-kube-api-access-lgrkr\") pod \"payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b\" (UID: \"54718bb1-f732-4c6b-aa42-7eda6ecec318\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b"
Oct 31 01:15:26.359281 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.359259    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq"]
Oct 31 01:15:26.360124 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.360104    4327 cpu_manager.go:414] "containerMap: removing container" podUID="79d8c2c4-a237-4c5b-9522-d64a581b918e" containerName="cleanup"
Oct 31 01:15:26.360124 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.360123    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d8c2c4-a237-4c5b-9522-d64a581b918e" containerName="cleanup"
Oct 31 01:15:26.360338 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.360255    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f3a0a27-f945-4566-9421-31d96a47f8f4" containerName="cleanup"
Oct 31 01:15:26.360338 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.360268    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d8c2c4-a237-4c5b-9522-d64a581b918e" containerName="cleanup"
Oct 31 01:15:26.360338 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.360276    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="25f9fe0d-fa0c-4b35-9cd3-33d2ded119b3" containerName="cleanup"
Oct 31 01:15:26.364305 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.364286    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq"
Oct 31 01:15:26.382704 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.382650    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq"]
Oct 31 01:15:26.460090 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.459962    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsmzc\" (UniqueName: \"kubernetes.io/projected/9ec0db43-a0c1-4e82-970b-dbe3f5585acd-kube-api-access-rsmzc\") pod \"payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq\" (UID: \"9ec0db43-a0c1-4e82-970b-dbe3f5585acd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq"
Oct 31 01:15:26.460090 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.460046    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec0db43-a0c1-4e82-970b-dbe3f5585acd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq\" (UID: \"9ec0db43-a0c1-4e82-970b-dbe3f5585acd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq"
Oct 31 01:15:26.460359 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.460122    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgrkr\" (UniqueName: \"kubernetes.io/projected/54718bb1-f732-4c6b-aa42-7eda6ecec318-kube-api-access-lgrkr\") pod \"payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b\" (UID: \"54718bb1-f732-4c6b-aa42-7eda6ecec318\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b"
Oct 31 01:15:26.460359 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.460332    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54718bb1-f732-4c6b-aa42-7eda6ecec318-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b\" (UID: \"54718bb1-f732-4c6b-aa42-7eda6ecec318\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b"
Oct 31 01:15:26.461283 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.461252    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54718bb1-f732-4c6b-aa42-7eda6ecec318-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b\" (UID: \"54718bb1-f732-4c6b-aa42-7eda6ecec318\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b"
Oct 31 01:15:26.484322 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.484293    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgrkr\" (UniqueName: \"kubernetes.io/projected/54718bb1-f732-4c6b-aa42-7eda6ecec318-kube-api-access-lgrkr\") pod \"payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b\" (UID: \"54718bb1-f732-4c6b-aa42-7eda6ecec318\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b"
Oct 31 01:15:26.562187 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.562141    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsmzc\" (UniqueName: \"kubernetes.io/projected/9ec0db43-a0c1-4e82-970b-dbe3f5585acd-kube-api-access-rsmzc\") pod \"payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq\" (UID: \"9ec0db43-a0c1-4e82-970b-dbe3f5585acd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq"
Oct 31 01:15:26.562376 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.562210    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec0db43-a0c1-4e82-970b-dbe3f5585acd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq\" (UID: \"9ec0db43-a0c1-4e82-970b-dbe3f5585acd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq"
Oct 31 01:15:26.562955 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.562934    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec0db43-a0c1-4e82-970b-dbe3f5585acd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq\" (UID: \"9ec0db43-a0c1-4e82-970b-dbe3f5585acd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq"
Oct 31 01:15:26.576711 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.576650    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b"
Oct 31 01:15:26.587471 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.587442    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsmzc\" (UniqueName: \"kubernetes.io/projected/9ec0db43-a0c1-4e82-970b-dbe3f5585acd-kube-api-access-rsmzc\") pod \"payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq\" (UID: \"9ec0db43-a0c1-4e82-970b-dbe3f5585acd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq"
Oct 31 01:15:26.679708 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.679181    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq"
Oct 31 01:15:26.811335 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.811284    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw" podUID="7aaf0c45-ac88-4c0f-b2a4-ab5539740af6" containerName="payload-processing" containerID="cri-o://a57932f6496a31b07617c27fede5614469cb6c10ee5f0ad3a0fda64bdb28e1b2" gracePeriod=25
Oct 31 01:15:26.855106 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.855060    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j" podUID="811136df-a3ab-4a3f-b826-8e7aea058eb8" containerName="payload-pre-processing" containerID="cri-o://51de4b0d5c1331177448422b586381d4c3d8527786ad33d2d0658207aedfa169" gracePeriod=25
Oct 31 01:15:26.914012 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:26.913983    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b"]
Oct 31 01:15:27.058540 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.058505    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq"]
Oct 31 01:15:27.263311 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.263278    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw"
Oct 31 01:15:27.277308 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.277262    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j"
Oct 31 01:15:27.278912 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.278880    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq" event={"ID":"9ec0db43-a0c1-4e82-970b-dbe3f5585acd","Type":"ContainerStarted","Data":"8dfacadceaa759c3a87bc15928442bf791a5038b24a6ab8ea4aa0caa5752b286"}
Oct 31 01:15:27.279065 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.278941    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq"
Oct 31 01:15:27.279065 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.278966    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq" event={"ID":"9ec0db43-a0c1-4e82-970b-dbe3f5585acd","Type":"ContainerStarted","Data":"ba2e2259bbed4d0260154c397e0d7a7ab4355b8753c4c61b2b069c30b92e1a3c"}
Oct 31 01:15:27.280501 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.280473    4327 generic.go:356] "Generic (PLEG): container finished" podID="7aaf0c45-ac88-4c0f-b2a4-ab5539740af6" containerID="a57932f6496a31b07617c27fede5614469cb6c10ee5f0ad3a0fda64bdb28e1b2" exitCode=0
Oct 31 01:15:27.280612 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.280584    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw" event={"ID":"7aaf0c45-ac88-4c0f-b2a4-ab5539740af6","Type":"ContainerDied","Data":"a57932f6496a31b07617c27fede5614469cb6c10ee5f0ad3a0fda64bdb28e1b2"}
Oct 31 01:15:27.280701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.280623    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw" event={"ID":"7aaf0c45-ac88-4c0f-b2a4-ab5539740af6","Type":"ContainerDied","Data":"aec780df60d2c44ed19e7eb617a5c0c106d92313a9a587e013714221313486ff"}
Oct 31 01:15:27.280701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.280655    4327 scope.go:122] "RemoveContainer" containerID="a57932f6496a31b07617c27fede5614469cb6c10ee5f0ad3a0fda64bdb28e1b2"
Oct 31 01:15:27.280830 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.280797    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw"
Oct 31 01:15:27.284324 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.284298    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b" event={"ID":"54718bb1-f732-4c6b-aa42-7eda6ecec318","Type":"ContainerStarted","Data":"0806ac6bb7767ffb554a440348ce5e3a6a1febab8a8c69709b47e4486f9dd4e9"}
Oct 31 01:15:27.284432 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.284346    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b" event={"ID":"54718bb1-f732-4c6b-aa42-7eda6ecec318","Type":"ContainerStarted","Data":"b5c67bdf5b236366b9878b2a704372bf10fb3f562f50ec1734215fb9d4e64bd5"}
Oct 31 01:15:27.284432 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.284400    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b"
Oct 31 01:15:27.289140 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.289117    4327 generic.go:356] "Generic (PLEG): container finished" podID="811136df-a3ab-4a3f-b826-8e7aea058eb8" containerID="51de4b0d5c1331177448422b586381d4c3d8527786ad33d2d0658207aedfa169" exitCode=0
Oct 31 01:15:27.289291 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.289270    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j"
Oct 31 01:15:27.289382 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.289295    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j" event={"ID":"811136df-a3ab-4a3f-b826-8e7aea058eb8","Type":"ContainerDied","Data":"51de4b0d5c1331177448422b586381d4c3d8527786ad33d2d0658207aedfa169"}
Oct 31 01:15:27.289382 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.289342    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j" event={"ID":"811136df-a3ab-4a3f-b826-8e7aea058eb8","Type":"ContainerDied","Data":"b35948f71875fa33b2c63cf73e75e06896be572bdf3beb9612b1455d9537f17c"}
Oct 31 01:15:27.300930 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.300903    4327 scope.go:122] "RemoveContainer" containerID="a57932f6496a31b07617c27fede5614469cb6c10ee5f0ad3a0fda64bdb28e1b2"
Oct 31 01:15:27.301221 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:15:27.301201    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a57932f6496a31b07617c27fede5614469cb6c10ee5f0ad3a0fda64bdb28e1b2\": container with ID starting with a57932f6496a31b07617c27fede5614469cb6c10ee5f0ad3a0fda64bdb28e1b2 not found: ID does not exist" containerID="a57932f6496a31b07617c27fede5614469cb6c10ee5f0ad3a0fda64bdb28e1b2"
Oct 31 01:15:27.301307 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.301225    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a57932f6496a31b07617c27fede5614469cb6c10ee5f0ad3a0fda64bdb28e1b2"} err="failed to get container status \"a57932f6496a31b07617c27fede5614469cb6c10ee5f0ad3a0fda64bdb28e1b2\": rpc error: code = NotFound desc = could not find container \"a57932f6496a31b07617c27fede5614469cb6c10ee5f0ad3a0fda64bdb28e1b2\": container with ID starting with a57932f6496a31b07617c27fede5614469cb6c10ee5f0ad3a0fda64bdb28e1b2 not found: ID does not exist"
Oct 31 01:15:27.301307 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.301241    4327 scope.go:122] "RemoveContainer" containerID="51de4b0d5c1331177448422b586381d4c3d8527786ad33d2d0658207aedfa169"
Oct 31 01:15:27.311871 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.311722    4327 scope.go:122] "RemoveContainer" containerID="51de4b0d5c1331177448422b586381d4c3d8527786ad33d2d0658207aedfa169"
Oct 31 01:15:27.312070 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:15:27.312048    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51de4b0d5c1331177448422b586381d4c3d8527786ad33d2d0658207aedfa169\": container with ID starting with 51de4b0d5c1331177448422b586381d4c3d8527786ad33d2d0658207aedfa169 not found: ID does not exist" containerID="51de4b0d5c1331177448422b586381d4c3d8527786ad33d2d0658207aedfa169"
Oct 31 01:15:27.312163 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.312083    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51de4b0d5c1331177448422b586381d4c3d8527786ad33d2d0658207aedfa169"} err="failed to get container status \"51de4b0d5c1331177448422b586381d4c3d8527786ad33d2d0658207aedfa169\": rpc error: code = NotFound desc = could not find container \"51de4b0d5c1331177448422b586381d4c3d8527786ad33d2d0658207aedfa169\": container with ID starting with 51de4b0d5c1331177448422b586381d4c3d8527786ad33d2d0658207aedfa169 not found: ID does not exist"
Oct 31 01:15:27.377889 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.377848    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aaf0c45-ac88-4c0f-b2a4-ab5539740af6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aaf0c45-ac88-4c0f-b2a4-ab5539740af6-plugins-config-volume\") pod \"7aaf0c45-ac88-4c0f-b2a4-ab5539740af6\" (UID: \"7aaf0c45-ac88-4c0f-b2a4-ab5539740af6\") "
Oct 31 01:15:27.378095 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.378001    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/811136df-a3ab-4a3f-b826-8e7aea058eb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/811136df-a3ab-4a3f-b826-8e7aea058eb8-plugins-config-volume\") pod \"811136df-a3ab-4a3f-b826-8e7aea058eb8\" (UID: \"811136df-a3ab-4a3f-b826-8e7aea058eb8\") "
Oct 31 01:15:27.378176 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.378108    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/811136df-a3ab-4a3f-b826-8e7aea058eb8-kube-api-access-7r7lc\" (UniqueName: \"kubernetes.io/projected/811136df-a3ab-4a3f-b826-8e7aea058eb8-kube-api-access-7r7lc\") pod \"811136df-a3ab-4a3f-b826-8e7aea058eb8\" (UID: \"811136df-a3ab-4a3f-b826-8e7aea058eb8\") "
Oct 31 01:15:27.378176 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.378155    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aaf0c45-ac88-4c0f-b2a4-ab5539740af6-kube-api-access-pwbdq\" (UniqueName: \"kubernetes.io/projected/7aaf0c45-ac88-4c0f-b2a4-ab5539740af6-kube-api-access-pwbdq\") pod \"7aaf0c45-ac88-4c0f-b2a4-ab5539740af6\" (UID: \"7aaf0c45-ac88-4c0f-b2a4-ab5539740af6\") "
Oct 31 01:15:27.379181 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.379152    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aaf0c45-ac88-4c0f-b2a4-ab5539740af6-plugins-config-volume" pod "7aaf0c45-ac88-4c0f-b2a4-ab5539740af6" (UID: "7aaf0c45-ac88-4c0f-b2a4-ab5539740af6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:27.379569 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.379541    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/811136df-a3ab-4a3f-b826-8e7aea058eb8-plugins-config-volume" pod "811136df-a3ab-4a3f-b826-8e7aea058eb8" (UID: "811136df-a3ab-4a3f-b826-8e7aea058eb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:27.382249 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.382223    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aaf0c45-ac88-4c0f-b2a4-ab5539740af6-kube-api-access-pwbdq" pod "7aaf0c45-ac88-4c0f-b2a4-ab5539740af6" (UID: "7aaf0c45-ac88-4c0f-b2a4-ab5539740af6"). InnerVolumeSpecName "kube-api-access-pwbdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:27.382359 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.382227    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811136df-a3ab-4a3f-b826-8e7aea058eb8-kube-api-access-7r7lc" pod "811136df-a3ab-4a3f-b826-8e7aea058eb8" (UID: "811136df-a3ab-4a3f-b826-8e7aea058eb8"). InnerVolumeSpecName "kube-api-access-7r7lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:27.453379 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.453297    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq" podStartSLOduration=1.45327491 podStartE2EDuration="1.45327491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:27.375466151 +0000 UTC m=+3387.597501695" watchObservedRunningTime="2026-10-31 01:15:27.45327491 +0000 UTC m=+3387.675310455"
Oct 31 01:15:27.480257 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.480214    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/811136df-a3ab-4a3f-b826-8e7aea058eb8-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:15:27.480257 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.480250    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r7lc\" (UniqueName: \"kubernetes.io/projected/811136df-a3ab-4a3f-b826-8e7aea058eb8-kube-api-access-7r7lc\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:15:27.480257 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.480263    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwbdq\" (UniqueName: \"kubernetes.io/projected/7aaf0c45-ac88-4c0f-b2a4-ab5539740af6-kube-api-access-pwbdq\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:15:27.480586 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.480276    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aaf0c45-ac88-4c0f-b2a4-ab5539740af6-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:15:27.516746 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.501782    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b" podStartSLOduration=1.501757923 podStartE2EDuration="1.501757923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:27.49313462 +0000 UTC m=+3387.715170163" watchObservedRunningTime="2026-10-31 01:15:27.501757923 +0000 UTC m=+3387.723793466"
Oct 31 01:15:27.687973 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.687929    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw"]
Oct 31 01:15:27.694641 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.694607    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-33d44f-b69895595-65qmw"]
Oct 31 01:15:27.700510 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.700398    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j"]
Oct 31 01:15:27.703743 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.703717    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-6485f5f65f-jnr6j"]
Oct 31 01:15:27.954669 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.954633    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aaf0c45-ac88-4c0f-b2a4-ab5539740af6" path="/var/lib/kubelet/pods/7aaf0c45-ac88-4c0f-b2a4-ab5539740af6/volumes"
Oct 31 01:15:27.955146 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:27.955132    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="811136df-a3ab-4a3f-b826-8e7aea058eb8" path="/var/lib/kubelet/pods/811136df-a3ab-4a3f-b826-8e7aea058eb8/volumes"
Oct 31 01:15:33.617572 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.617528    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main" containerID="cri-o://6c40379f8bbb6dd7b412610d94d5047a961890861a904e5bcfec28aaf7411142" gracePeriod=45
Oct 31 01:15:33.651351 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.651301    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main" containerID="cri-o://fc317fe8176d758f56bae6909ee2bb84fe588da7e3f6dc6eae6e147fda9ab8ba" gracePeriod=45
Oct 31 01:15:33.946001 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.945973    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 01:15:33.950931 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.950904    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 01:15:33.984460 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.984425    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-home\" (UniqueName: \"kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-home\") pod \"b6a8824b-19f2-47df-88e5-442724a36100\" (UID: \"b6a8824b-19f2-47df-88e5-442724a36100\") "
Oct 31 01:15:33.984650 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.984488    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-tmp-dir\") pod \"b6a8824b-19f2-47df-88e5-442724a36100\" (UID: \"b6a8824b-19f2-47df-88e5-442724a36100\") "
Oct 31 01:15:33.984650 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.984521    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6a8824b-19f2-47df-88e5-442724a36100-tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a8824b-19f2-47df-88e5-442724a36100-tls-certs\") pod \"b6a8824b-19f2-47df-88e5-442724a36100\" (UID: \"b6a8824b-19f2-47df-88e5-442724a36100\") "
Oct 31 01:15:33.984650 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.984553    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-dshm\") pod \"b6a8824b-19f2-47df-88e5-442724a36100\" (UID: \"b6a8824b-19f2-47df-88e5-442724a36100\") "
Oct 31 01:15:33.984650 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.984624    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-kube-api-access-xt2gx\" (UniqueName: \"kubernetes.io/projected/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-kube-api-access-xt2gx\") pod \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\" (UID: \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\") "
Oct 31 01:15:33.984910 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.984660    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-tls-certs\" (UniqueName: \"kubernetes.io/secret/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-tls-certs\") pod \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\" (UID: \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\") "
Oct 31 01:15:33.984910 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.984768    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-home" pod "b6a8824b-19f2-47df-88e5-442724a36100" (UID: "b6a8824b-19f2-47df-88e5-442724a36100"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:15:33.984910 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.984778    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-tmp-dir" pod "b6a8824b-19f2-47df-88e5-442724a36100" (UID: "b6a8824b-19f2-47df-88e5-442724a36100"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:15:33.984910 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.984775    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-home\" (UniqueName: \"kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-home\") pod \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\" (UID: \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\") "
Oct 31 01:15:33.984910 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.984869    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-tmp-dir\") pod \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\" (UID: \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\") "
Oct 31 01:15:33.985199 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.984911    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-dshm\") pod \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\" (UID: \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\") "
Oct 31 01:15:33.985199 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.984999    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-home" pod "e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" (UID: "e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:15:33.985199 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.985025    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-model-cache\") pod \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\" (UID: \"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de\") "
Oct 31 01:15:33.985199 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.985121    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-model-cache\") pod \"b6a8824b-19f2-47df-88e5-442724a36100\" (UID: \"b6a8824b-19f2-47df-88e5-442724a36100\") "
Oct 31 01:15:33.985436 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.985169    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-tmp-dir" pod "e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" (UID: "e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:15:33.985436 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.985183    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6a8824b-19f2-47df-88e5-442724a36100-kube-api-access-pk566\" (UniqueName: \"kubernetes.io/projected/b6a8824b-19f2-47df-88e5-442724a36100-kube-api-access-pk566\") pod \"b6a8824b-19f2-47df-88e5-442724a36100\" (UID: \"b6a8824b-19f2-47df-88e5-442724a36100\") "
Oct 31 01:15:33.985436 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.985251    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-model-cache" pod "e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" (UID: "e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:15:33.985436 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.985424    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-model-cache" pod "b6a8824b-19f2-47df-88e5-442724a36100" (UID: "b6a8824b-19f2-47df-88e5-442724a36100"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:15:33.986300 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.986215    4327 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-home\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:15:33.986300 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.986245    4327 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-tmp-dir\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:15:33.986300 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.986260    4327 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-model-cache\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:15:33.986300 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.986272    4327 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-model-cache\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:15:33.986300 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.986284    4327 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-home\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:15:33.986300 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.986296    4327 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-tmp-dir\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:15:33.987310 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.987275    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-tls-certs" pod "e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" (UID: "e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:15:33.987421 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.987372    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-dshm" pod "b6a8824b-19f2-47df-88e5-442724a36100" (UID: "b6a8824b-19f2-47df-88e5-442724a36100"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:15:33.987503 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.987479    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6a8824b-19f2-47df-88e5-442724a36100-tls-certs" pod "b6a8824b-19f2-47df-88e5-442724a36100" (UID: "b6a8824b-19f2-47df-88e5-442724a36100"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:15:33.987566 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.987544    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-kube-api-access-xt2gx" pod "e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" (UID: "e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de"). InnerVolumeSpecName "kube-api-access-xt2gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:33.987601 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.987566    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-dshm" pod "e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" (UID: "e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:15:33.988413 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:33.988392    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a8824b-19f2-47df-88e5-442724a36100-kube-api-access-pk566" pod "b6a8824b-19f2-47df-88e5-442724a36100" (UID: "b6a8824b-19f2-47df-88e5-442724a36100"). InnerVolumeSpecName "kube-api-access-pk566". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:34.088343 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.088306    4327 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a8824b-19f2-47df-88e5-442724a36100-dshm\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:15:34.088343 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.088346    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt2gx\" (UniqueName: \"kubernetes.io/projected/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-kube-api-access-xt2gx\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:15:34.088555 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.088363    4327 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-tls-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:15:34.088555 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.088378    4327 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de-dshm\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:15:34.088555 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.088392    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk566\" (UniqueName: \"kubernetes.io/projected/b6a8824b-19f2-47df-88e5-442724a36100-kube-api-access-pk566\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:15:34.088555 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.088404    4327 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a8824b-19f2-47df-88e5-442724a36100-tls-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:15:34.348992 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.348897    4327 generic.go:356] "Generic (PLEG): container finished" podID="cae467a1-6a9a-4c1b-855a-ac8edb1d10ee" containerID="30ab02190365330996820b709bd42274b2e961ee11f70d487aa8a90bcb827acd" exitCode=0
Oct 31 01:15:34.348992 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.348979    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2" event={"ID":"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee","Type":"ContainerDied","Data":"30ab02190365330996820b709bd42274b2e961ee11f70d487aa8a90bcb827acd"}
Oct 31 01:15:34.349494 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.349475    4327 scope.go:122] "RemoveContainer" containerID="30ab02190365330996820b709bd42274b2e961ee11f70d487aa8a90bcb827acd"
Oct 31 01:15:34.350525 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.350504    4327 generic.go:356] "Generic (PLEG): container finished" podID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerID="fc317fe8176d758f56bae6909ee2bb84fe588da7e3f6dc6eae6e147fda9ab8ba" exitCode=0
Oct 31 01:15:34.350613 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.350534    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" event={"ID":"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de","Type":"ContainerDied","Data":"fc317fe8176d758f56bae6909ee2bb84fe588da7e3f6dc6eae6e147fda9ab8ba"}
Oct 31 01:15:34.350613 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.350559    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"
Oct 31 01:15:34.350613 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.350574    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp" event={"ID":"e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de","Type":"ContainerDied","Data":"188457dac756358d6626a35d22f4c298cfb2e1d20f77adee5ea5703cbb5c610f"}
Oct 31 01:15:34.350613 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.350594    4327 scope.go:122] "RemoveContainer" containerID="fc317fe8176d758f56bae6909ee2bb84fe588da7e3f6dc6eae6e147fda9ab8ba"
Oct 31 01:15:34.351775 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.351739    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:15:34.351950 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.351838    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:15:34.351950 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.351888    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:15:34.352255 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.352223    4327 generic.go:356] "Generic (PLEG): container finished" podID="b6a8824b-19f2-47df-88e5-442724a36100" containerID="6c40379f8bbb6dd7b412610d94d5047a961890861a904e5bcfec28aaf7411142" exitCode=0
Oct 31 01:15:34.352358 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.352256    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"
Oct 31 01:15:34.352358 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.352290    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" event={"ID":"b6a8824b-19f2-47df-88e5-442724a36100","Type":"ContainerDied","Data":"6c40379f8bbb6dd7b412610d94d5047a961890861a904e5bcfec28aaf7411142"}
Oct 31 01:15:34.352358 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.352314    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8" event={"ID":"b6a8824b-19f2-47df-88e5-442724a36100","Type":"ContainerDied","Data":"e0cbe8cd5cbfd2d9b956f40302ea288a3a3169708eea915e4ce63f0817053d65"}
Oct 31 01:15:34.364045 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.364024    4327 scope.go:122] "RemoveContainer" containerID="fc317fe8176d758f56bae6909ee2bb84fe588da7e3f6dc6eae6e147fda9ab8ba"
Oct 31 01:15:34.364415 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:15:34.364386    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc317fe8176d758f56bae6909ee2bb84fe588da7e3f6dc6eae6e147fda9ab8ba\": container with ID starting with fc317fe8176d758f56bae6909ee2bb84fe588da7e3f6dc6eae6e147fda9ab8ba not found: ID does not exist" containerID="fc317fe8176d758f56bae6909ee2bb84fe588da7e3f6dc6eae6e147fda9ab8ba"
Oct 31 01:15:34.364492 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.364424    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc317fe8176d758f56bae6909ee2bb84fe588da7e3f6dc6eae6e147fda9ab8ba"} err="failed to get container status \"fc317fe8176d758f56bae6909ee2bb84fe588da7e3f6dc6eae6e147fda9ab8ba\": rpc error: code = NotFound desc = could not find container \"fc317fe8176d758f56bae6909ee2bb84fe588da7e3f6dc6eae6e147fda9ab8ba\": container with ID starting with fc317fe8176d758f56bae6909ee2bb84fe588da7e3f6dc6eae6e147fda9ab8ba not found: ID does not exist"
Oct 31 01:15:34.364492 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.364448    4327 scope.go:122] "RemoveContainer" containerID="6c40379f8bbb6dd7b412610d94d5047a961890861a904e5bcfec28aaf7411142"
Oct 31 01:15:34.374220 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.374199    4327 scope.go:122] "RemoveContainer" containerID="6c40379f8bbb6dd7b412610d94d5047a961890861a904e5bcfec28aaf7411142"
Oct 31 01:15:34.374571 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:15:34.374552    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c40379f8bbb6dd7b412610d94d5047a961890861a904e5bcfec28aaf7411142\": container with ID starting with 6c40379f8bbb6dd7b412610d94d5047a961890861a904e5bcfec28aaf7411142 not found: ID does not exist" containerID="6c40379f8bbb6dd7b412610d94d5047a961890861a904e5bcfec28aaf7411142"
Oct 31 01:15:34.374624 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.374579    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c40379f8bbb6dd7b412610d94d5047a961890861a904e5bcfec28aaf7411142"} err="failed to get container status \"6c40379f8bbb6dd7b412610d94d5047a961890861a904e5bcfec28aaf7411142\": rpc error: code = NotFound desc = could not find container \"6c40379f8bbb6dd7b412610d94d5047a961890861a904e5bcfec28aaf7411142\": container with ID starting with 6c40379f8bbb6dd7b412610d94d5047a961890861a904e5bcfec28aaf7411142 not found: ID does not exist"
Oct 31 01:15:34.455391 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.455313    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"]
Oct 31 01:15:34.460024 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.459993    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0d8bae/premium-w4-0d8bae-kserve-89f6c45cd-gxfpp"]
Oct 31 01:15:34.500010 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.499973    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"]
Oct 31 01:15:34.503368 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:34.503346    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0d8bae/distinct-2-w4-0d8bae-kserve-864dc6f798-8pfs8"]
Oct 31 01:15:35.361425 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:35.361385    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2" event={"ID":"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee","Type":"ContainerStarted","Data":"edc647a438c1f9c18824aaf21ef6d30242b5a60a8b9d6ac65a20a68dc06649d6"}
Oct 31 01:15:35.783412 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:35.783316    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:15:35.784851 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:35.784823    4327 patch_prober.go:29] interesting pod/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 31 01:15:35.784959 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:35.784885    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2" podUID="cae467a1-6a9a-4c1b-855a-ac8edb1d10ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 31 01:15:35.944804 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:35.944751    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6a8824b-19f2-47df-88e5-442724a36100" path="/var/lib/kubelet/pods/b6a8824b-19f2-47df-88e5-442724a36100/volumes"
Oct 31 01:15:35.945372 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:35.945353    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" path="/var/lib/kubelet/pods/e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de/volumes"
Oct 31 01:15:36.783402 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:36.783366    4327 patch_prober.go:29] interesting pod/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 31 01:15:36.783882 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:36.783419    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2" podUID="cae467a1-6a9a-4c1b-855a-ac8edb1d10ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 31 01:15:37.783251 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:37.783210    4327 patch_prober.go:29] interesting pod/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 31 01:15:37.783471 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:37.783280    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2" podUID="cae467a1-6a9a-4c1b-855a-ac8edb1d10ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 31 01:15:38.150391 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:38.150355    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:15:38.301810 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:38.301777    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq"
Oct 31 01:15:38.302267 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:38.302245    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b"
Oct 31 01:15:38.783852 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:38.783813    4327 patch_prober.go:29] interesting pod/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 31 01:15:38.784274 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:38.783873    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2" podUID="cae467a1-6a9a-4c1b-855a-ac8edb1d10ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 31 01:15:39.783196 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:39.783162    4327 patch_prober.go:29] interesting pod/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 31 01:15:39.783388 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:39.783215    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2" podUID="cae467a1-6a9a-4c1b-855a-ac8edb1d10ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 31 01:15:40.783425 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:40.783389    4327 patch_prober.go:29] interesting pod/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 31 01:15:40.783896 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:40.783445    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2" podUID="cae467a1-6a9a-4c1b-855a-ac8edb1d10ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 31 01:15:41.787821 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:41.787780    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:15:41.788920 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:41.788897    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:15:45.294807 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:15:45.294768    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:15:49.477023 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:49.476990    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d68696-884f5657-t256p"]
Oct 31 01:15:49.477774 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:49.477749    4327 cpu_manager.go:414] "containerMap: removing container" podUID="7aaf0c45-ac88-4c0f-b2a4-ab5539740af6" containerName="payload-processing"
Oct 31 01:15:49.477774 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:49.477766    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aaf0c45-ac88-4c0f-b2a4-ab5539740af6" containerName="payload-processing"
Oct 31 01:15:49.477984 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:49.477783    4327 cpu_manager.go:414] "containerMap: removing container" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main"
Oct 31 01:15:49.477984 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:49.477790    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main"
Oct 31 01:15:49.477984 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:49.477803    4327 cpu_manager.go:414] "containerMap: removing container" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main"
Oct 31 01:15:49.477984 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:49.477809    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main"
Oct 31 01:15:49.477984 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:49.477830    4327 cpu_manager.go:414] "containerMap: removing container" podUID="811136df-a3ab-4a3f-b826-8e7aea058eb8" containerName="payload-pre-processing"
Oct 31 01:15:49.477984 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:49.477835    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="811136df-a3ab-4a3f-b826-8e7aea058eb8" containerName="payload-pre-processing"
Oct 31 01:15:49.477984 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:49.477950    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aaf0c45-ac88-4c0f-b2a4-ab5539740af6" containerName="payload-processing"
Oct 31 01:15:49.477984 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:49.477959    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6a8824b-19f2-47df-88e5-442724a36100" containerName="main"
Oct 31 01:15:49.477984 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:49.477966    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4cee987-bfc0-4a16-9ae7-1e9eaf42b3de" containerName="main"
Oct 31 01:15:49.477984 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:49.477975    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="811136df-a3ab-4a3f-b826-8e7aea058eb8" containerName="payload-pre-processing"
Oct 31 01:15:49.482512 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:49.482492    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-884f5657-t256p"
Oct 31 01:15:49.504108 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:49.504084    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d68696\""
Oct 31 01:15:49.520370 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:49.520340    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d68696-884f5657-t256p"]
Oct 31 01:15:49.638701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:49.637712    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/165d92f7-faab-45a9-97e6-4f90ecd1cbd2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d68696-884f5657-t256p\" (UID: \"165d92f7-faab-45a9-97e6-4f90ecd1cbd2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-884f5657-t256p"
Oct 31 01:15:49.638701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:49.637891    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bpgt\" (UniqueName: \"kubernetes.io/projected/165d92f7-faab-45a9-97e6-4f90ecd1cbd2-kube-api-access-8bpgt\") pod \"payload-processing-e2e-ipp-b-d68696-884f5657-t256p\" (UID: \"165d92f7-faab-45a9-97e6-4f90ecd1cbd2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-884f5657-t256p"
Oct 31 01:15:49.739950 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:49.739861    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/165d92f7-faab-45a9-97e6-4f90ecd1cbd2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d68696-884f5657-t256p\" (UID: \"165d92f7-faab-45a9-97e6-4f90ecd1cbd2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-884f5657-t256p"
Oct 31 01:15:49.740115 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:49.740007    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bpgt\" (UniqueName: \"kubernetes.io/projected/165d92f7-faab-45a9-97e6-4f90ecd1cbd2-kube-api-access-8bpgt\") pod \"payload-processing-e2e-ipp-b-d68696-884f5657-t256p\" (UID: \"165d92f7-faab-45a9-97e6-4f90ecd1cbd2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-884f5657-t256p"
Oct 31 01:15:49.740592 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:49.740562    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/165d92f7-faab-45a9-97e6-4f90ecd1cbd2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d68696-884f5657-t256p\" (UID: \"165d92f7-faab-45a9-97e6-4f90ecd1cbd2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-884f5657-t256p"
Oct 31 01:15:49.783529 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:49.783490    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bpgt\" (UniqueName: \"kubernetes.io/projected/165d92f7-faab-45a9-97e6-4f90ecd1cbd2-kube-api-access-8bpgt\") pod \"payload-processing-e2e-ipp-b-d68696-884f5657-t256p\" (UID: \"165d92f7-faab-45a9-97e6-4f90ecd1cbd2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-884f5657-t256p"
Oct 31 01:15:49.797407 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:49.797375    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-884f5657-t256p"
Oct 31 01:15:50.124809 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:15:50.124774    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod165d92f7_faab_45a9_97e6_4f90ecd1cbd2.slice/crio-6782004dc67fa07ddea91905920fdd1f96d7baef58c892a280404147141ffd66 WatchSource:0}: Error finding container 6782004dc67fa07ddea91905920fdd1f96d7baef58c892a280404147141ffd66: Status 404 returned error can't find the container with id 6782004dc67fa07ddea91905920fdd1f96d7baef58c892a280404147141ffd66
Oct 31 01:15:50.134544 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:50.134516    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d68696-884f5657-t256p"]
Oct 31 01:15:50.481645 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:50.481546    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-884f5657-t256p" event={"ID":"165d92f7-faab-45a9-97e6-4f90ecd1cbd2","Type":"ContainerStarted","Data":"ae12daa0248aa1fd2bfcaa6deb8990f138c3dd80b8ddadf2ec48f5b69d7a0605"}
Oct 31 01:15:50.481645 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:50.481603    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-884f5657-t256p" event={"ID":"165d92f7-faab-45a9-97e6-4f90ecd1cbd2","Type":"ContainerStarted","Data":"6782004dc67fa07ddea91905920fdd1f96d7baef58c892a280404147141ffd66"}
Oct 31 01:15:50.481645 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:50.481626    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-884f5657-t256p"
Oct 31 01:15:50.525615 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:50.525558    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-884f5657-t256p" podStartSLOduration=1.525543366 podStartE2EDuration="1.525543366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:50.522380264 +0000 UTC m=+3410.744415805" watchObservedRunningTime="2026-10-31 01:15:50.525543366 +0000 UTC m=+3410.747578908"
Oct 31 01:15:50.784109 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:15:50.784031    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"]
Oct 31 01:16:01.494535 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:01.494506    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-884f5657-t256p"
Oct 31 01:16:03.261425 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:03.261386    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt"]
Oct 31 01:16:03.267482 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:03.267444    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt"
Oct 31 01:16:03.292562 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:03.292525    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt"]
Oct 31 01:16:03.300885 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:03.300854    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-33d44f-dockercfg-2r5fm\""
Oct 31 01:16:03.301900 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:03.301846    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-33d44f\""
Oct 31 01:16:03.462258 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:03.462220    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw9p8\" (UniqueName: \"kubernetes.io/projected/782150a7-2f48-4aac-9e16-8c3d6cb53a82-kube-api-access-bw9p8\") pod \"payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt\" (UID: \"782150a7-2f48-4aac-9e16-8c3d6cb53a82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt"
Oct 31 01:16:03.462660 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:03.462633    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/782150a7-2f48-4aac-9e16-8c3d6cb53a82-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt\" (UID: \"782150a7-2f48-4aac-9e16-8c3d6cb53a82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt"
Oct 31 01:16:03.563576 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:03.563490    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw9p8\" (UniqueName: \"kubernetes.io/projected/782150a7-2f48-4aac-9e16-8c3d6cb53a82-kube-api-access-bw9p8\") pod \"payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt\" (UID: \"782150a7-2f48-4aac-9e16-8c3d6cb53a82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt"
Oct 31 01:16:03.563766 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:03.563646    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/782150a7-2f48-4aac-9e16-8c3d6cb53a82-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt\" (UID: \"782150a7-2f48-4aac-9e16-8c3d6cb53a82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt"
Oct 31 01:16:03.564369 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:03.564348    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/782150a7-2f48-4aac-9e16-8c3d6cb53a82-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt\" (UID: \"782150a7-2f48-4aac-9e16-8c3d6cb53a82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt"
Oct 31 01:16:03.587794 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:03.587761    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw9p8\" (UniqueName: \"kubernetes.io/projected/782150a7-2f48-4aac-9e16-8c3d6cb53a82-kube-api-access-bw9p8\") pod \"payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt\" (UID: \"782150a7-2f48-4aac-9e16-8c3d6cb53a82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt"
Oct 31 01:16:03.609018 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:03.608989    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt"
Oct 31 01:16:03.929623 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:16:03.929589    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod782150a7_2f48_4aac_9e16_8c3d6cb53a82.slice/crio-a557c5e9b59d4c7c7653c52371db37b6508445e4b41875fd5aa2699dec05f9bb WatchSource:0}: Error finding container a557c5e9b59d4c7c7653c52371db37b6508445e4b41875fd5aa2699dec05f9bb: Status 404 returned error can't find the container with id a557c5e9b59d4c7c7653c52371db37b6508445e4b41875fd5aa2699dec05f9bb
Oct 31 01:16:03.930349 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:03.930327    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt"]
Oct 31 01:16:04.594224 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:04.594122    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt" event={"ID":"782150a7-2f48-4aac-9e16-8c3d6cb53a82","Type":"ContainerStarted","Data":"18959641a1a8b047f2e24030e0f6f2a4c4ad5ac220cf61549162458e4f8cfc83"}
Oct 31 01:16:04.594224 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:04.594217    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt"
Oct 31 01:16:04.594721 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:04.594236    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt" event={"ID":"782150a7-2f48-4aac-9e16-8c3d6cb53a82","Type":"ContainerStarted","Data":"a557c5e9b59d4c7c7653c52371db37b6508445e4b41875fd5aa2699dec05f9bb"}
Oct 31 01:16:04.660174 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:04.660120    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt" podStartSLOduration=1.660105847 podStartE2EDuration="1.660105847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:04.646471497 +0000 UTC m=+3424.868507039" watchObservedRunningTime="2026-10-31 01:16:04.660105847 +0000 UTC m=+3424.882141388"
Oct 31 01:16:05.806403 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:05.806343    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb" podUID="c65a86d1-1074-4f4c-95db-c7b26378938f" containerName="main" containerID="cri-o://3ab4197ab1b276032d060128e1147697bb2ceabc867b7bbca9ec2729f79335dd" gracePeriod=45
Oct 31 01:16:05.885006 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:05.884962    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb" podUID="c65a86d1-1074-4f4c-95db-c7b26378938f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.202:8000/ready\": dial tcp 10.133.0.202:8000: connect: connection refused"
Oct 31 01:16:06.134362 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.134334    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"
Oct 31 01:16:06.207071 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.207039    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c65a86d1-1074-4f4c-95db-c7b26378938f-tls-certs\" (UniqueName: \"kubernetes.io/secret/c65a86d1-1074-4f4c-95db-c7b26378938f-tls-certs\") pod \"c65a86d1-1074-4f4c-95db-c7b26378938f\" (UID: \"c65a86d1-1074-4f4c-95db-c7b26378938f\") "
Oct 31 01:16:06.207266 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.207244    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c65a86d1-1074-4f4c-95db-c7b26378938f-kube-api-access-qclcp\" (UniqueName: \"kubernetes.io/projected/c65a86d1-1074-4f4c-95db-c7b26378938f-kube-api-access-qclcp\") pod \"c65a86d1-1074-4f4c-95db-c7b26378938f\" (UID: \"c65a86d1-1074-4f4c-95db-c7b26378938f\") "
Oct 31 01:16:06.207355 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.207339    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-tmp-dir\") pod \"c65a86d1-1074-4f4c-95db-c7b26378938f\" (UID: \"c65a86d1-1074-4f4c-95db-c7b26378938f\") "
Oct 31 01:16:06.207421 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.207375    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-model-cache\") pod \"c65a86d1-1074-4f4c-95db-c7b26378938f\" (UID: \"c65a86d1-1074-4f4c-95db-c7b26378938f\") "
Oct 31 01:16:06.207494 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.207465    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-dshm\") pod \"c65a86d1-1074-4f4c-95db-c7b26378938f\" (UID: \"c65a86d1-1074-4f4c-95db-c7b26378938f\") "
Oct 31 01:16:06.207667 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.207586    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-home\" (UniqueName: \"kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-home\") pod \"c65a86d1-1074-4f4c-95db-c7b26378938f\" (UID: \"c65a86d1-1074-4f4c-95db-c7b26378938f\") "
Oct 31 01:16:06.207877 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.207712    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-model-cache" pod "c65a86d1-1074-4f4c-95db-c7b26378938f" (UID: "c65a86d1-1074-4f4c-95db-c7b26378938f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:06.207877 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.207852    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-home" pod "c65a86d1-1074-4f4c-95db-c7b26378938f" (UID: "c65a86d1-1074-4f4c-95db-c7b26378938f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:06.208052 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.207949    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-tmp-dir" pod "c65a86d1-1074-4f4c-95db-c7b26378938f" (UID: "c65a86d1-1074-4f4c-95db-c7b26378938f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:06.208342 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.208318    4327 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-home\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:16:06.208446 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.208349    4327 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-tmp-dir\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:16:06.208446 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.208365    4327 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-model-cache\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:16:06.209731 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.209704    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-dshm" pod "c65a86d1-1074-4f4c-95db-c7b26378938f" (UID: "c65a86d1-1074-4f4c-95db-c7b26378938f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:06.209848 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.209756    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c65a86d1-1074-4f4c-95db-c7b26378938f-kube-api-access-qclcp" pod "c65a86d1-1074-4f4c-95db-c7b26378938f" (UID: "c65a86d1-1074-4f4c-95db-c7b26378938f"). InnerVolumeSpecName "kube-api-access-qclcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:16:06.209848 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.209796    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c65a86d1-1074-4f4c-95db-c7b26378938f-tls-certs" pod "c65a86d1-1074-4f4c-95db-c7b26378938f" (UID: "c65a86d1-1074-4f4c-95db-c7b26378938f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:16:06.309414 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.309382    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qclcp\" (UniqueName: \"kubernetes.io/projected/c65a86d1-1074-4f4c-95db-c7b26378938f-kube-api-access-qclcp\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:16:06.309414 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.309412    4327 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c65a86d1-1074-4f4c-95db-c7b26378938f-dshm\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:16:06.309414 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.309423    4327 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c65a86d1-1074-4f4c-95db-c7b26378938f-tls-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:16:06.610660 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.610627    4327 generic.go:356] "Generic (PLEG): container finished" podID="c65a86d1-1074-4f4c-95db-c7b26378938f" containerID="3ab4197ab1b276032d060128e1147697bb2ceabc867b7bbca9ec2729f79335dd" exitCode=0
Oct 31 01:16:06.610867 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.610674    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"
Oct 31 01:16:06.610867 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.610712    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb" event={"ID":"c65a86d1-1074-4f4c-95db-c7b26378938f","Type":"ContainerDied","Data":"3ab4197ab1b276032d060128e1147697bb2ceabc867b7bbca9ec2729f79335dd"}
Oct 31 01:16:06.610867 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.610763    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb" event={"ID":"c65a86d1-1074-4f4c-95db-c7b26378938f","Type":"ContainerDied","Data":"98c219a0a7c0c300314b1ee4a157d799175b4e15c3d81561b6eb30c3286f1ff5"}
Oct 31 01:16:06.610867 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.610787    4327 scope.go:122] "RemoveContainer" containerID="3ab4197ab1b276032d060128e1147697bb2ceabc867b7bbca9ec2729f79335dd"
Oct 31 01:16:06.621877 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.621856    4327 scope.go:122] "RemoveContainer" containerID="3ab4197ab1b276032d060128e1147697bb2ceabc867b7bbca9ec2729f79335dd"
Oct 31 01:16:06.622142 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:16:06.622122    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ab4197ab1b276032d060128e1147697bb2ceabc867b7bbca9ec2729f79335dd\": container with ID starting with 3ab4197ab1b276032d060128e1147697bb2ceabc867b7bbca9ec2729f79335dd not found: ID does not exist" containerID="3ab4197ab1b276032d060128e1147697bb2ceabc867b7bbca9ec2729f79335dd"
Oct 31 01:16:06.622213 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.622153    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ab4197ab1b276032d060128e1147697bb2ceabc867b7bbca9ec2729f79335dd"} err="failed to get container status \"3ab4197ab1b276032d060128e1147697bb2ceabc867b7bbca9ec2729f79335dd\": rpc error: code = NotFound desc = could not find container \"3ab4197ab1b276032d060128e1147697bb2ceabc867b7bbca9ec2729f79335dd\": container with ID starting with 3ab4197ab1b276032d060128e1147697bb2ceabc867b7bbca9ec2729f79335dd not found: ID does not exist"
Oct 31 01:16:06.670721 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.670672    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"]
Oct 31 01:16:06.681870 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:06.681837    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d10721/auth-test-model-d10721-kserve-5d57899cfc-gkskb"]
Oct 31 01:16:07.923642 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:07.923608    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c65a86d1-1074-4f4c-95db-c7b26378938f" path="/var/lib/kubelet/pods/c65a86d1-1074-4f4c-95db-c7b26378938f/volumes"
Oct 31 01:16:15.295173 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:16:15.295129    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:16:15.604529 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:15.604495    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt"
Oct 31 01:16:21.139240 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:21.139200    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"]
Oct 31 01:16:36.164428 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.164380    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5" podUID="a283db50-3ca7-494a-8cac-6653efff5251" containerName="main" containerID="cri-o://b3ef79d5d73681c7f729b85a4f4bf440242de56392e5e0188c17966b913670e9" gracePeriod=45
Oct 31 01:16:36.458010 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.457976    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"
Oct 31 01:16:36.595357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.595322    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-dshm\") pod \"a283db50-3ca7-494a-8cac-6653efff5251\" (UID: \"a283db50-3ca7-494a-8cac-6653efff5251\") "
Oct 31 01:16:36.595673 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.595382    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-tmp-dir\") pod \"a283db50-3ca7-494a-8cac-6653efff5251\" (UID: \"a283db50-3ca7-494a-8cac-6653efff5251\") "
Oct 31 01:16:36.595673 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.595432    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a283db50-3ca7-494a-8cac-6653efff5251-kube-api-access-9pfbp\" (UniqueName: \"kubernetes.io/projected/a283db50-3ca7-494a-8cac-6653efff5251-kube-api-access-9pfbp\") pod \"a283db50-3ca7-494a-8cac-6653efff5251\" (UID: \"a283db50-3ca7-494a-8cac-6653efff5251\") "
Oct 31 01:16:36.595673 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.595517    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-model-cache\") pod \"a283db50-3ca7-494a-8cac-6653efff5251\" (UID: \"a283db50-3ca7-494a-8cac-6653efff5251\") "
Oct 31 01:16:36.595673 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.595545    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-home\" (UniqueName: \"kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-home\") pod \"a283db50-3ca7-494a-8cac-6653efff5251\" (UID: \"a283db50-3ca7-494a-8cac-6653efff5251\") "
Oct 31 01:16:36.595942 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.595726    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-tmp-dir" pod "a283db50-3ca7-494a-8cac-6653efff5251" (UID: "a283db50-3ca7-494a-8cac-6653efff5251"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:36.595942 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.595831    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a283db50-3ca7-494a-8cac-6653efff5251-tls-certs\" (UniqueName: \"kubernetes.io/secret/a283db50-3ca7-494a-8cac-6653efff5251-tls-certs\") pod \"a283db50-3ca7-494a-8cac-6653efff5251\" (UID: \"a283db50-3ca7-494a-8cac-6653efff5251\") "
Oct 31 01:16:36.595942 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.595864    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-model-cache" pod "a283db50-3ca7-494a-8cac-6653efff5251" (UID: "a283db50-3ca7-494a-8cac-6653efff5251"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:36.595942 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.595926    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-home" pod "a283db50-3ca7-494a-8cac-6653efff5251" (UID: "a283db50-3ca7-494a-8cac-6653efff5251"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:36.596365 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.596343    4327 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-tmp-dir\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:16:36.596365 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.596366    4327 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-model-cache\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:16:36.596502 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.596375    4327 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-home\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:16:36.597817 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.597792    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-dshm" pod "a283db50-3ca7-494a-8cac-6653efff5251" (UID: "a283db50-3ca7-494a-8cac-6653efff5251"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:36.597817 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.597809    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a283db50-3ca7-494a-8cac-6653efff5251-kube-api-access-9pfbp" pod "a283db50-3ca7-494a-8cac-6653efff5251" (UID: "a283db50-3ca7-494a-8cac-6653efff5251"). InnerVolumeSpecName "kube-api-access-9pfbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:16:36.597946 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.597855    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a283db50-3ca7-494a-8cac-6653efff5251-tls-certs" pod "a283db50-3ca7-494a-8cac-6653efff5251" (UID: "a283db50-3ca7-494a-8cac-6653efff5251"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:16:36.698259 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.698173    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pfbp\" (UniqueName: \"kubernetes.io/projected/a283db50-3ca7-494a-8cac-6653efff5251-kube-api-access-9pfbp\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:16:36.698259 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.698204    4327 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a283db50-3ca7-494a-8cac-6653efff5251-tls-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:16:36.698259 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.698214    4327 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a283db50-3ca7-494a-8cac-6653efff5251-dshm\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:16:36.848797 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.848756    4327 generic.go:356] "Generic (PLEG): container finished" podID="a283db50-3ca7-494a-8cac-6653efff5251" containerID="b3ef79d5d73681c7f729b85a4f4bf440242de56392e5e0188c17966b913670e9" exitCode=0
Oct 31 01:16:36.848973 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.848820    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"
Oct 31 01:16:36.848973 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.848833    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5" event={"ID":"a283db50-3ca7-494a-8cac-6653efff5251","Type":"ContainerDied","Data":"b3ef79d5d73681c7f729b85a4f4bf440242de56392e5e0188c17966b913670e9"}
Oct 31 01:16:36.848973 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.848878    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5" event={"ID":"a283db50-3ca7-494a-8cac-6653efff5251","Type":"ContainerDied","Data":"3d9e37edeecd950a31e254f2d2ece3b1b0a4628d71bc028ccc557d8c9c0cc219"}
Oct 31 01:16:36.848973 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.848901    4327 scope.go:122] "RemoveContainer" containerID="b3ef79d5d73681c7f729b85a4f4bf440242de56392e5e0188c17966b913670e9"
Oct 31 01:16:36.861650 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.861623    4327 scope.go:122] "RemoveContainer" containerID="b3ef79d5d73681c7f729b85a4f4bf440242de56392e5e0188c17966b913670e9"
Oct 31 01:16:36.862076 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:16:36.862049    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3ef79d5d73681c7f729b85a4f4bf440242de56392e5e0188c17966b913670e9\": container with ID starting with b3ef79d5d73681c7f729b85a4f4bf440242de56392e5e0188c17966b913670e9 not found: ID does not exist" containerID="b3ef79d5d73681c7f729b85a4f4bf440242de56392e5e0188c17966b913670e9"
Oct 31 01:16:36.862144 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.862090    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3ef79d5d73681c7f729b85a4f4bf440242de56392e5e0188c17966b913670e9"} err="failed to get container status \"b3ef79d5d73681c7f729b85a4f4bf440242de56392e5e0188c17966b913670e9\": rpc error: code = NotFound desc = could not find container \"b3ef79d5d73681c7f729b85a4f4bf440242de56392e5e0188c17966b913670e9\": container with ID starting with b3ef79d5d73681c7f729b85a4f4bf440242de56392e5e0188c17966b913670e9 not found: ID does not exist"
Oct 31 01:16:36.903075 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.903043    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"]
Oct 31 01:16:36.908502 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:36.908476    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ada0c1/auth-test-model-ada0c1-kserve-9fbf9f569-5zjq5"]
Oct 31 01:16:37.561099 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:37.561064    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc"]
Oct 31 01:16:37.562072 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:37.561336    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc" podUID="f39647e7-efed-4224-80d9-e7a1cfc9c8a1" containerName="maas-api" containerID="cri-o://821112320cef9412c0dcf8734d65bb1ec9ef2d04fde70e7b5ca04fcd02080ea6" gracePeriod=30
Oct 31 01:16:37.874216 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:37.874183    4327 generic.go:356] "Generic (PLEG): container finished" podID="f39647e7-efed-4224-80d9-e7a1cfc9c8a1" containerID="821112320cef9412c0dcf8734d65bb1ec9ef2d04fde70e7b5ca04fcd02080ea6" exitCode=0
Oct 31 01:16:37.874387 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:37.874364    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc" event={"ID":"f39647e7-efed-4224-80d9-e7a1cfc9c8a1","Type":"ContainerDied","Data":"821112320cef9412c0dcf8734d65bb1ec9ef2d04fde70e7b5ca04fcd02080ea6"}
Oct 31 01:16:37.894667 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:37.894635    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc"
Oct 31 01:16:37.924605 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:37.924573    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a283db50-3ca7-494a-8cac-6653efff5251" path="/var/lib/kubelet/pods/a283db50-3ca7-494a-8cac-6653efff5251/volumes"
Oct 31 01:16:37.961706 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:37.957804    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-kube-api-access-4gv6h\" (UniqueName: \"kubernetes.io/projected/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-kube-api-access-4gv6h\") pod \"f39647e7-efed-4224-80d9-e7a1cfc9c8a1\" (UID: \"f39647e7-efed-4224-80d9-e7a1cfc9c8a1\") "
Oct 31 01:16:37.969728 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:37.966138    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-openshift-service-ca\") pod \"f39647e7-efed-4224-80d9-e7a1cfc9c8a1\" (UID: \"f39647e7-efed-4224-80d9-e7a1cfc9c8a1\") "
Oct 31 01:16:37.987708 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:37.986896    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-openshift-service-ca" pod "f39647e7-efed-4224-80d9-e7a1cfc9c8a1" (UID: "f39647e7-efed-4224-80d9-e7a1cfc9c8a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:16:37.987708 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:37.987108    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-maas-api-tls\") pod \"f39647e7-efed-4224-80d9-e7a1cfc9c8a1\" (UID: \"f39647e7-efed-4224-80d9-e7a1cfc9c8a1\") "
Oct 31 01:16:37.987708 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:37.987208    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-metrics-certs\") pod \"f39647e7-efed-4224-80d9-e7a1cfc9c8a1\" (UID: \"f39647e7-efed-4224-80d9-e7a1cfc9c8a1\") "
Oct 31 01:16:37.991700 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:37.988238    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:16:38.029116 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:38.019841    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-metrics-certs" pod "f39647e7-efed-4224-80d9-e7a1cfc9c8a1" (UID: "f39647e7-efed-4224-80d9-e7a1cfc9c8a1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:16:38.029116 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:38.020384    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-maas-api-tls" pod "f39647e7-efed-4224-80d9-e7a1cfc9c8a1" (UID: "f39647e7-efed-4224-80d9-e7a1cfc9c8a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:16:38.060535 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:38.060500    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-kube-api-access-4gv6h" pod "f39647e7-efed-4224-80d9-e7a1cfc9c8a1" (UID: "f39647e7-efed-4224-80d9-e7a1cfc9c8a1"). InnerVolumeSpecName "kube-api-access-4gv6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:16:38.096164 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:38.094574    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:16:38.096164 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:38.094673    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gv6h\" (UniqueName: \"kubernetes.io/projected/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-kube-api-access-4gv6h\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:16:38.096164 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:38.094705    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f39647e7-efed-4224-80d9-e7a1cfc9c8a1-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:16:38.885605 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:38.885567    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc"
Oct 31 01:16:38.886134 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:38.885572    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc" event={"ID":"f39647e7-efed-4224-80d9-e7a1cfc9c8a1","Type":"ContainerDied","Data":"bd372d48f529ed8865623aa11f38fa421248f12d0f95ce9c8fc3d3598f7814d7"}
Oct 31 01:16:38.886134 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:38.885776    4327 scope.go:122] "RemoveContainer" containerID="821112320cef9412c0dcf8734d65bb1ec9ef2d04fde70e7b5ca04fcd02080ea6"
Oct 31 01:16:38.939425 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:38.939385    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc"]
Oct 31 01:16:38.951856 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:38.951813    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0d8bae-86b59bc489-wg8tc"]
Oct 31 01:16:39.925050 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:39.925020    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f39647e7-efed-4224-80d9-e7a1cfc9c8a1" path="/var/lib/kubelet/pods/f39647e7-efed-4224-80d9-e7a1cfc9c8a1/volumes"
Oct 31 01:16:40.349121 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.349018    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8"]
Oct 31 01:16:40.350165 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.350137    4327 cpu_manager.go:414] "containerMap: removing container" podUID="a283db50-3ca7-494a-8cac-6653efff5251" containerName="main"
Oct 31 01:16:40.350165 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.350163    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a283db50-3ca7-494a-8cac-6653efff5251" containerName="main"
Oct 31 01:16:40.350455 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.350219    4327 cpu_manager.go:414] "containerMap: removing container" podUID="f39647e7-efed-4224-80d9-e7a1cfc9c8a1" containerName="maas-api"
Oct 31 01:16:40.350455 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.350227    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f39647e7-efed-4224-80d9-e7a1cfc9c8a1" containerName="maas-api"
Oct 31 01:16:40.350455 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.350253    4327 cpu_manager.go:414] "containerMap: removing container" podUID="c65a86d1-1074-4f4c-95db-c7b26378938f" containerName="main"
Oct 31 01:16:40.350455 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.350260    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c65a86d1-1074-4f4c-95db-c7b26378938f" containerName="main"
Oct 31 01:16:40.350455 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.350436    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="f39647e7-efed-4224-80d9-e7a1cfc9c8a1" containerName="maas-api"
Oct 31 01:16:40.350455 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.350449    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="c65a86d1-1074-4f4c-95db-c7b26378938f" containerName="main"
Oct 31 01:16:40.350754 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.350463    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="a283db50-3ca7-494a-8cac-6653efff5251" containerName="main"
Oct 31 01:16:40.361496 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.360752    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8"
Oct 31 01:16:40.374918 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.374883    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8"]
Oct 31 01:16:40.383001 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.382969    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fa97ea\""
Oct 31 01:16:40.383001 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.382985    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fa97ea-dockercfg-gq2qx\""
Oct 31 01:16:40.441338 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.441298    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e576a213-4352-4cea-b68a-388edabf4ab0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8\" (UID: \"e576a213-4352-4cea-b68a-388edabf4ab0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8"
Oct 31 01:16:40.441859 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.441463    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm5rz\" (UniqueName: \"kubernetes.io/projected/e576a213-4352-4cea-b68a-388edabf4ab0-kube-api-access-mm5rz\") pod \"payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8\" (UID: \"e576a213-4352-4cea-b68a-388edabf4ab0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8"
Oct 31 01:16:40.525509 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.525475    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv"]
Oct 31 01:16:40.530221 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.530197    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv"
Oct 31 01:16:40.543063 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.543031    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e576a213-4352-4cea-b68a-388edabf4ab0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8\" (UID: \"e576a213-4352-4cea-b68a-388edabf4ab0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8"
Oct 31 01:16:40.543222 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.543123    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm5rz\" (UniqueName: \"kubernetes.io/projected/e576a213-4352-4cea-b68a-388edabf4ab0-kube-api-access-mm5rz\") pod \"payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8\" (UID: \"e576a213-4352-4cea-b68a-388edabf4ab0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8"
Oct 31 01:16:40.543721 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.543667    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e576a213-4352-4cea-b68a-388edabf4ab0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8\" (UID: \"e576a213-4352-4cea-b68a-388edabf4ab0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8"
Oct 31 01:16:40.551442 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.551412    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv"]
Oct 31 01:16:40.589525 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.589498    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm5rz\" (UniqueName: \"kubernetes.io/projected/e576a213-4352-4cea-b68a-388edabf4ab0-kube-api-access-mm5rz\") pod \"payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8\" (UID: \"e576a213-4352-4cea-b68a-388edabf4ab0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8"
Oct 31 01:16:40.644425 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.644386    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gmkr\" (UniqueName: \"kubernetes.io/projected/2c2c3784-1773-40a2-b373-f925c7e4d239-kube-api-access-7gmkr\") pod \"payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv\" (UID: \"2c2c3784-1773-40a2-b373-f925c7e4d239\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv"
Oct 31 01:16:40.644666 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.644606    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c2c3784-1773-40a2-b373-f925c7e4d239-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv\" (UID: \"2c2c3784-1773-40a2-b373-f925c7e4d239\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv"
Oct 31 01:16:40.687327 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.687299    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8"
Oct 31 01:16:40.745784 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.745748    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c2c3784-1773-40a2-b373-f925c7e4d239-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv\" (UID: \"2c2c3784-1773-40a2-b373-f925c7e4d239\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv"
Oct 31 01:16:40.745918 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.745898    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gmkr\" (UniqueName: \"kubernetes.io/projected/2c2c3784-1773-40a2-b373-f925c7e4d239-kube-api-access-7gmkr\") pod \"payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv\" (UID: \"2c2c3784-1773-40a2-b373-f925c7e4d239\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv"
Oct 31 01:16:40.746524 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.746501    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c2c3784-1773-40a2-b373-f925c7e4d239-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv\" (UID: \"2c2c3784-1773-40a2-b373-f925c7e4d239\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv"
Oct 31 01:16:40.792003 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.782651    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gmkr\" (UniqueName: \"kubernetes.io/projected/2c2c3784-1773-40a2-b373-f925c7e4d239-kube-api-access-7gmkr\") pod \"payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv\" (UID: \"2c2c3784-1773-40a2-b373-f925c7e4d239\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv"
Oct 31 01:16:40.845251 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.845212    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv"
Oct 31 01:16:40.992718 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:16:40.992671    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode576a213_4352_4cea_b68a_388edabf4ab0.slice/crio-67a2a62e785e24c0f4bb86412a90847edfdfaea6b14ab62cf97bed5f7baa9385 WatchSource:0}: Error finding container 67a2a62e785e24c0f4bb86412a90847edfdfaea6b14ab62cf97bed5f7baa9385: Status 404 returned error can't find the container with id 67a2a62e785e24c0f4bb86412a90847edfdfaea6b14ab62cf97bed5f7baa9385
Oct 31 01:16:40.993061 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:40.992736    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8"]
Oct 31 01:16:41.155185 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:16:41.155148    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2c3784_1773_40a2_b373_f925c7e4d239.slice/crio-8a252ad83fe3f02972041550d07bbba250724b5d5f1e49941b0078770c9cd214 WatchSource:0}: Error finding container 8a252ad83fe3f02972041550d07bbba250724b5d5f1e49941b0078770c9cd214: Status 404 returned error can't find the container with id 8a252ad83fe3f02972041550d07bbba250724b5d5f1e49941b0078770c9cd214
Oct 31 01:16:41.155788 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:41.155722    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv"]
Oct 31 01:16:41.910549 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:41.910506    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv" event={"ID":"2c2c3784-1773-40a2-b373-f925c7e4d239","Type":"ContainerStarted","Data":"a649def0154655d5bf2311c48de1bb7457eb5d7f726de8fb66f12be3ccdc23d2"}
Oct 31 01:16:41.910800 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:41.910576    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv" event={"ID":"2c2c3784-1773-40a2-b373-f925c7e4d239","Type":"ContainerStarted","Data":"8a252ad83fe3f02972041550d07bbba250724b5d5f1e49941b0078770c9cd214"}
Oct 31 01:16:41.910800 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:41.910610    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv"
Oct 31 01:16:41.912313 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:41.912288    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8" event={"ID":"e576a213-4352-4cea-b68a-388edabf4ab0","Type":"ContainerStarted","Data":"00bd701edc5adae6316d7017d11770333ab84c71b7ae2aa617ac636400ea1ab2"}
Oct 31 01:16:41.912440 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:41.912336    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8" event={"ID":"e576a213-4352-4cea-b68a-388edabf4ab0","Type":"ContainerStarted","Data":"67a2a62e785e24c0f4bb86412a90847edfdfaea6b14ab62cf97bed5f7baa9385"}
Oct 31 01:16:41.912440 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:41.912362    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8"
Oct 31 01:16:41.955869 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:41.955806    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv" podStartSLOduration=1.955792411 podStartE2EDuration="1.955792411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:41.954057272 +0000 UTC m=+3462.176092824" watchObservedRunningTime="2026-10-31 01:16:41.955792411 +0000 UTC m=+3462.177827944"
Oct 31 01:16:41.997639 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:41.997580    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8" podStartSLOduration=1.997564337 podStartE2EDuration="1.997564337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:41.996847258 +0000 UTC m=+3462.218882827" watchObservedRunningTime="2026-10-31 01:16:41.997564337 +0000 UTC m=+3462.219599879"
Oct 31 01:16:45.294721 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:16:45.294651    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:16:52.921895 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:52.921859    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8"
Oct 31 01:16:52.922384 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:16:52.922282    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv"
Oct 31 01:17:02.818621 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:02.818581    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7"]
Oct 31 01:17:02.826289 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:02.826211    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7"
Oct 31 01:17:02.836394 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:02.836363    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7"]
Oct 31 01:17:02.863757 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:02.863728    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8d251f\""
Oct 31 01:17:02.870988 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:02.870961    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8d251f-dockercfg-wkhjv\""
Oct 31 01:17:03.016138 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:03.016102    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k27tl\" (UniqueName: \"kubernetes.io/projected/c1501512-ab3e-4907-880d-5b1f0bb68946-kube-api-access-k27tl\") pod \"payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7\" (UID: \"c1501512-ab3e-4907-880d-5b1f0bb68946\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7"
Oct 31 01:17:03.016366 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:03.016333    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1501512-ab3e-4907-880d-5b1f0bb68946-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7\" (UID: \"c1501512-ab3e-4907-880d-5b1f0bb68946\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7"
Oct 31 01:17:03.118029 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:03.117991    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1501512-ab3e-4907-880d-5b1f0bb68946-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7\" (UID: \"c1501512-ab3e-4907-880d-5b1f0bb68946\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7"
Oct 31 01:17:03.118229 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:03.118112    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k27tl\" (UniqueName: \"kubernetes.io/projected/c1501512-ab3e-4907-880d-5b1f0bb68946-kube-api-access-k27tl\") pod \"payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7\" (UID: \"c1501512-ab3e-4907-880d-5b1f0bb68946\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7"
Oct 31 01:17:03.118981 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:03.118956    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1501512-ab3e-4907-880d-5b1f0bb68946-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7\" (UID: \"c1501512-ab3e-4907-880d-5b1f0bb68946\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7"
Oct 31 01:17:03.142113 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:03.142079    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k27tl\" (UniqueName: \"kubernetes.io/projected/c1501512-ab3e-4907-880d-5b1f0bb68946-kube-api-access-k27tl\") pod \"payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7\" (UID: \"c1501512-ab3e-4907-880d-5b1f0bb68946\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7"
Oct 31 01:17:03.142890 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:03.142871    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7"
Oct 31 01:17:03.451577 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:17:03.451546    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1501512_ab3e_4907_880d_5b1f0bb68946.slice/crio-528af64d82097dcdd10973a38f4d7a332424498b6e8543b1cda4966b61104871 WatchSource:0}: Error finding container 528af64d82097dcdd10973a38f4d7a332424498b6e8543b1cda4966b61104871: Status 404 returned error can't find the container with id 528af64d82097dcdd10973a38f4d7a332424498b6e8543b1cda4966b61104871
Oct 31 01:17:03.452709 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:03.452639    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7"]
Oct 31 01:17:04.141181 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:04.141139    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7" event={"ID":"c1501512-ab3e-4907-880d-5b1f0bb68946","Type":"ContainerStarted","Data":"b6fdc4da777ff56b5d0de8b544767b70c47a7712d788eedfb51087f48fe49368"}
Oct 31 01:17:04.141732 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:04.141204    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7" event={"ID":"c1501512-ab3e-4907-880d-5b1f0bb68946","Type":"ContainerStarted","Data":"528af64d82097dcdd10973a38f4d7a332424498b6e8543b1cda4966b61104871"}
Oct 31 01:17:04.141732 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:04.141235    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7"
Oct 31 01:17:04.189929 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:04.189866    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7" podStartSLOduration=2.189846699 podStartE2EDuration="2.189846699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:04.188011516 +0000 UTC m=+3484.410047057" watchObservedRunningTime="2026-10-31 01:17:04.189846699 +0000 UTC m=+3484.411882313"
Oct 31 01:17:05.846477 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:05.846445    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"]
Oct 31 01:17:05.853282 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:05.853244    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"
Oct 31 01:17:05.871793 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:05.871742    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"]
Oct 31 01:17:05.881671 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:05.881640    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-ada0c1\"/\"kube-root-ca.crt\""
Oct 31 01:17:05.881882 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:05.881733    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-ada0c1\"/\"default-dockercfg-627sx\""
Oct 31 01:17:05.881882 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:05.881843    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-ada0c1\"/\"rate-test-model-ada0c1-kserve-self-signed-certs\""
Oct 31 01:17:05.882009 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:05.881892    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-ada0c1\"/\"openshift-service-ca.crt\""
Oct 31 01:17:05.892890 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:05.892856    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnrst\" (UniqueName: \"kubernetes.io/projected/08107827-859a-452d-9d7e-85377ee4b3ab-kube-api-access-gnrst\") pod \"rate-test-model-ada0c1-kserve-5985cf6565-qslxz\" (UID: \"08107827-859a-452d-9d7e-85377ee4b3ab\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"
Oct 31 01:17:05.893085 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:05.892924    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-model-cache\") pod \"rate-test-model-ada0c1-kserve-5985cf6565-qslxz\" (UID: \"08107827-859a-452d-9d7e-85377ee4b3ab\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"
Oct 31 01:17:05.893172 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:05.893083    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-tmp-dir\") pod \"rate-test-model-ada0c1-kserve-5985cf6565-qslxz\" (UID: \"08107827-859a-452d-9d7e-85377ee4b3ab\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"
Oct 31 01:17:05.893172 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:05.893131    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08107827-859a-452d-9d7e-85377ee4b3ab-tls-certs\") pod \"rate-test-model-ada0c1-kserve-5985cf6565-qslxz\" (UID: \"08107827-859a-452d-9d7e-85377ee4b3ab\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"
Oct 31 01:17:05.893344 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:05.893320    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-home\") pod \"rate-test-model-ada0c1-kserve-5985cf6565-qslxz\" (UID: \"08107827-859a-452d-9d7e-85377ee4b3ab\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"
Oct 31 01:17:05.893393 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:05.893369    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-dshm\") pod \"rate-test-model-ada0c1-kserve-5985cf6565-qslxz\" (UID: \"08107827-859a-452d-9d7e-85377ee4b3ab\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"
Oct 31 01:17:05.994820 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:05.994784    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-tmp-dir\") pod \"rate-test-model-ada0c1-kserve-5985cf6565-qslxz\" (UID: \"08107827-859a-452d-9d7e-85377ee4b3ab\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"
Oct 31 01:17:05.995037 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:05.994929    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08107827-859a-452d-9d7e-85377ee4b3ab-tls-certs\") pod \"rate-test-model-ada0c1-kserve-5985cf6565-qslxz\" (UID: \"08107827-859a-452d-9d7e-85377ee4b3ab\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"
Oct 31 01:17:05.995115 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:05.995078    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-home\") pod \"rate-test-model-ada0c1-kserve-5985cf6565-qslxz\" (UID: \"08107827-859a-452d-9d7e-85377ee4b3ab\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"
Oct 31 01:17:05.995173 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:05.995149    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-dshm\") pod \"rate-test-model-ada0c1-kserve-5985cf6565-qslxz\" (UID: \"08107827-859a-452d-9d7e-85377ee4b3ab\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"
Oct 31 01:17:05.995274 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:05.995250    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-tmp-dir\") pod \"rate-test-model-ada0c1-kserve-5985cf6565-qslxz\" (UID: \"08107827-859a-452d-9d7e-85377ee4b3ab\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"
Oct 31 01:17:05.995347 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:05.995316    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnrst\" (UniqueName: \"kubernetes.io/projected/08107827-859a-452d-9d7e-85377ee4b3ab-kube-api-access-gnrst\") pod \"rate-test-model-ada0c1-kserve-5985cf6565-qslxz\" (UID: \"08107827-859a-452d-9d7e-85377ee4b3ab\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"
Oct 31 01:17:05.995441 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:05.995416    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-home\") pod \"rate-test-model-ada0c1-kserve-5985cf6565-qslxz\" (UID: \"08107827-859a-452d-9d7e-85377ee4b3ab\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"
Oct 31 01:17:05.995504 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:05.995414    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-model-cache\") pod \"rate-test-model-ada0c1-kserve-5985cf6565-qslxz\" (UID: \"08107827-859a-452d-9d7e-85377ee4b3ab\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"
Oct 31 01:17:05.995841 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:05.995808    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-model-cache\") pod \"rate-test-model-ada0c1-kserve-5985cf6565-qslxz\" (UID: \"08107827-859a-452d-9d7e-85377ee4b3ab\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"
Oct 31 01:17:05.997600 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:05.997572    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08107827-859a-452d-9d7e-85377ee4b3ab-tls-certs\") pod \"rate-test-model-ada0c1-kserve-5985cf6565-qslxz\" (UID: \"08107827-859a-452d-9d7e-85377ee4b3ab\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"
Oct 31 01:17:05.997600 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:05.997572    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-dshm\") pod \"rate-test-model-ada0c1-kserve-5985cf6565-qslxz\" (UID: \"08107827-859a-452d-9d7e-85377ee4b3ab\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"
Oct 31 01:17:06.019558 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:06.019530    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnrst\" (UniqueName: \"kubernetes.io/projected/08107827-859a-452d-9d7e-85377ee4b3ab-kube-api-access-gnrst\") pod \"rate-test-model-ada0c1-kserve-5985cf6565-qslxz\" (UID: \"08107827-859a-452d-9d7e-85377ee4b3ab\") " pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"
Oct 31 01:17:06.172143 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:06.172108    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"
Oct 31 01:17:06.497054 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:06.497023    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"]
Oct 31 01:17:06.499288 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:17:06.499260    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08107827_859a_452d_9d7e_85377ee4b3ab.slice/crio-a25c4b7470046275db384eb29fcc7183d6ce551fd9c959c02c8099f8e190d73c WatchSource:0}: Error finding container a25c4b7470046275db384eb29fcc7183d6ce551fd9c959c02c8099f8e190d73c: Status 404 returned error can't find the container with id a25c4b7470046275db384eb29fcc7183d6ce551fd9c959c02c8099f8e190d73c
Oct 31 01:17:07.177426 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:07.177391    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz" event={"ID":"08107827-859a-452d-9d7e-85377ee4b3ab","Type":"ContainerStarted","Data":"6e019a6222b063cbcb151b19cd61c80d86999c1ac711672cca9b7270f9d220e8"}
Oct 31 01:17:07.177426 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:07.177433    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz" event={"ID":"08107827-859a-452d-9d7e-85377ee4b3ab","Type":"ContainerStarted","Data":"a25c4b7470046275db384eb29fcc7183d6ce551fd9c959c02c8099f8e190d73c"}
Oct 31 01:17:07.223134 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:07.223081    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz" podStartSLOduration=2.223066964 podStartE2EDuration="2.223066964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:07.219528902 +0000 UTC m=+3487.441564443" watchObservedRunningTime="2026-10-31 01:17:07.223066964 +0000 UTC m=+3487.445102506"
Oct 31 01:17:08.172722 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:08.172674    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"
Oct 31 01:17:15.162365 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:15.162323    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7"
Oct 31 01:17:15.300749 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:17:15.300667    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:17:16.172492 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:16.172455    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"
Oct 31 01:17:16.186109 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:16.186080    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"
Oct 31 01:17:16.318557 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:16.318530    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"
Oct 31 01:17:17.645291 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:17.645251    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t"]
Oct 31 01:17:17.645851 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:17.645516    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t" podUID="fc74ef25-794f-4410-b3b7-403c7ba9f7d8" containerName="maas-api" containerID="cri-o://e3d33b14d748db76712767ff0171bbd9aeb76962dd305f51ccb35a9b074468aa" gracePeriod=30
Oct 31 01:17:17.799274 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:17.799242    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57"]
Oct 31 01:17:17.799468 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:17.799447    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57" podUID="7900b962-7b7a-4d5e-b78f-8d8fe95d3123" containerName="payload-processing" containerID="cri-o://820ae26d7dfeb0a7438f67557585735e4d601a74434620dfb6628a48a7112a25" gracePeriod=30
Oct 31 01:17:17.833283 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:17.833247    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h"]
Oct 31 01:17:17.833790 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:17.833756    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h" podUID="5c54b47b-3540-45b9-8ac7-8b1f056b0b81" containerName="payload-pre-processing" containerID="cri-o://ea13a63ccd5acd13266ebf8430f643cee616cd2a3cd83fb2e3a89e5b62a80297" gracePeriod=30
Oct 31 01:17:18.018663 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.018631    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t"
Oct 31 01:17:18.128608 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.128544    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57"
Oct 31 01:17:18.177804 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.176868    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-maas-api-tls\") pod \"fc74ef25-794f-4410-b3b7-403c7ba9f7d8\" (UID: \"fc74ef25-794f-4410-b3b7-403c7ba9f7d8\") "
Oct 31 01:17:18.177804 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.176999    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-kube-api-access-5c4kz\" (UniqueName: \"kubernetes.io/projected/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-kube-api-access-5c4kz\") pod \"fc74ef25-794f-4410-b3b7-403c7ba9f7d8\" (UID: \"fc74ef25-794f-4410-b3b7-403c7ba9f7d8\") "
Oct 31 01:17:18.177804 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.177086    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-metrics-certs\") pod \"fc74ef25-794f-4410-b3b7-403c7ba9f7d8\" (UID: \"fc74ef25-794f-4410-b3b7-403c7ba9f7d8\") "
Oct 31 01:17:18.177804 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.177121    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-openshift-service-ca\") pod \"fc74ef25-794f-4410-b3b7-403c7ba9f7d8\" (UID: \"fc74ef25-794f-4410-b3b7-403c7ba9f7d8\") "
Oct 31 01:17:18.178146 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.177962    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-openshift-service-ca" pod "fc74ef25-794f-4410-b3b7-403c7ba9f7d8" (UID: "fc74ef25-794f-4410-b3b7-403c7ba9f7d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:17:18.179765 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.179735    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-metrics-certs" pod "fc74ef25-794f-4410-b3b7-403c7ba9f7d8" (UID: "fc74ef25-794f-4410-b3b7-403c7ba9f7d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:17:18.179917 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.179783    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-maas-api-tls" pod "fc74ef25-794f-4410-b3b7-403c7ba9f7d8" (UID: "fc74ef25-794f-4410-b3b7-403c7ba9f7d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:17:18.180208 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.180181    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-kube-api-access-5c4kz" pod "fc74ef25-794f-4410-b3b7-403c7ba9f7d8" (UID: "fc74ef25-794f-4410-b3b7-403c7ba9f7d8"). InnerVolumeSpecName "kube-api-access-5c4kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:17:18.181727 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.181706    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h"
Oct 31 01:17:18.278991 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.278941    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c54b47b-3540-45b9-8ac7-8b1f056b0b81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c54b47b-3540-45b9-8ac7-8b1f056b0b81-plugins-config-volume\") pod \"5c54b47b-3540-45b9-8ac7-8b1f056b0b81\" (UID: \"5c54b47b-3540-45b9-8ac7-8b1f056b0b81\") "
Oct 31 01:17:18.279324 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.279285    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7900b962-7b7a-4d5e-b78f-8d8fe95d3123-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7900b962-7b7a-4d5e-b78f-8d8fe95d3123-plugins-config-volume\") pod \"7900b962-7b7a-4d5e-b78f-8d8fe95d3123\" (UID: \"7900b962-7b7a-4d5e-b78f-8d8fe95d3123\") "
Oct 31 01:17:18.279845 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.279799    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c54b47b-3540-45b9-8ac7-8b1f056b0b81-kube-api-access-nfjg2\" (UniqueName: \"kubernetes.io/projected/5c54b47b-3540-45b9-8ac7-8b1f056b0b81-kube-api-access-nfjg2\") pod \"5c54b47b-3540-45b9-8ac7-8b1f056b0b81\" (UID: \"5c54b47b-3540-45b9-8ac7-8b1f056b0b81\") "
Oct 31 01:17:18.280068 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.280050    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7900b962-7b7a-4d5e-b78f-8d8fe95d3123-kube-api-access-2fmc6\" (UniqueName: \"kubernetes.io/projected/7900b962-7b7a-4d5e-b78f-8d8fe95d3123-kube-api-access-2fmc6\") pod \"7900b962-7b7a-4d5e-b78f-8d8fe95d3123\" (UID: \"7900b962-7b7a-4d5e-b78f-8d8fe95d3123\") "
Oct 31 01:17:18.280995 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.280974    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:17:18.281148 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.281131    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c4kz\" (UniqueName: \"kubernetes.io/projected/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-kube-api-access-5c4kz\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:17:18.281271 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.281258    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:17:18.281386 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.281374    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc74ef25-794f-4410-b3b7-403c7ba9f7d8-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:17:18.282352 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.282201    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7900b962-7b7a-4d5e-b78f-8d8fe95d3123-plugins-config-volume" pod "7900b962-7b7a-4d5e-b78f-8d8fe95d3123" (UID: "7900b962-7b7a-4d5e-b78f-8d8fe95d3123"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:17:18.282967 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.282849    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c54b47b-3540-45b9-8ac7-8b1f056b0b81-plugins-config-volume" pod "5c54b47b-3540-45b9-8ac7-8b1f056b0b81" (UID: "5c54b47b-3540-45b9-8ac7-8b1f056b0b81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:17:18.336490 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.336428    4327 generic.go:356] "Generic (PLEG): container finished" podID="7900b962-7b7a-4d5e-b78f-8d8fe95d3123" containerID="820ae26d7dfeb0a7438f67557585735e4d601a74434620dfb6628a48a7112a25" exitCode=0
Oct 31 01:17:18.337103 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.337065    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57" event={"ID":"7900b962-7b7a-4d5e-b78f-8d8fe95d3123","Type":"ContainerDied","Data":"820ae26d7dfeb0a7438f67557585735e4d601a74434620dfb6628a48a7112a25"}
Oct 31 01:17:18.338722 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.338692    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57" event={"ID":"7900b962-7b7a-4d5e-b78f-8d8fe95d3123","Type":"ContainerDied","Data":"3f8c4bda9c599036082e14084f10c3b67c2a4e128d90fe6f41674b55e75f8db1"}
Oct 31 01:17:18.338964 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.338947    4327 scope.go:122] "RemoveContainer" containerID="820ae26d7dfeb0a7438f67557585735e4d601a74434620dfb6628a48a7112a25"
Oct 31 01:17:18.339212 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.338047    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57"
Oct 31 01:17:18.341524 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.341177    4327 generic.go:356] "Generic (PLEG): container finished" podID="5c54b47b-3540-45b9-8ac7-8b1f056b0b81" containerID="ea13a63ccd5acd13266ebf8430f643cee616cd2a3cd83fb2e3a89e5b62a80297" exitCode=0
Oct 31 01:17:18.341524 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.341301    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h" event={"ID":"5c54b47b-3540-45b9-8ac7-8b1f056b0b81","Type":"ContainerDied","Data":"ea13a63ccd5acd13266ebf8430f643cee616cd2a3cd83fb2e3a89e5b62a80297"}
Oct 31 01:17:18.341524 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.341335    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h" event={"ID":"5c54b47b-3540-45b9-8ac7-8b1f056b0b81","Type":"ContainerDied","Data":"c5a7bc6aee9b7c3fe8e9364995fbd5e95bb3f2579ed5fb3f95a6d975c4af9973"}
Oct 31 01:17:18.341524 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.341397    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h"
Oct 31 01:17:18.344016 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.343313    4327 generic.go:356] "Generic (PLEG): container finished" podID="fc74ef25-794f-4410-b3b7-403c7ba9f7d8" containerID="e3d33b14d748db76712767ff0171bbd9aeb76962dd305f51ccb35a9b074468aa" exitCode=0
Oct 31 01:17:18.344016 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.343393    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t" event={"ID":"fc74ef25-794f-4410-b3b7-403c7ba9f7d8","Type":"ContainerDied","Data":"e3d33b14d748db76712767ff0171bbd9aeb76962dd305f51ccb35a9b074468aa"}
Oct 31 01:17:18.344016 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.343421    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t" event={"ID":"fc74ef25-794f-4410-b3b7-403c7ba9f7d8","Type":"ContainerDied","Data":"31e0e42208af52853fbd2e4888298f39874ee8fcc3c7582421f7fe2fea0f6a5e"}
Oct 31 01:17:18.344016 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.343477    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t"
Oct 31 01:17:18.358108 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.358068    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7900b962-7b7a-4d5e-b78f-8d8fe95d3123-kube-api-access-2fmc6" pod "7900b962-7b7a-4d5e-b78f-8d8fe95d3123" (UID: "7900b962-7b7a-4d5e-b78f-8d8fe95d3123"). InnerVolumeSpecName "kube-api-access-2fmc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:17:18.361469 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.359512    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c54b47b-3540-45b9-8ac7-8b1f056b0b81-kube-api-access-nfjg2" pod "5c54b47b-3540-45b9-8ac7-8b1f056b0b81" (UID: "5c54b47b-3540-45b9-8ac7-8b1f056b0b81"). InnerVolumeSpecName "kube-api-access-nfjg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:17:18.368602 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.368579    4327 scope.go:122] "RemoveContainer" containerID="820ae26d7dfeb0a7438f67557585735e4d601a74434620dfb6628a48a7112a25"
Oct 31 01:17:18.370645 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:17:18.370011    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"820ae26d7dfeb0a7438f67557585735e4d601a74434620dfb6628a48a7112a25\": container with ID starting with 820ae26d7dfeb0a7438f67557585735e4d601a74434620dfb6628a48a7112a25 not found: ID does not exist" containerID="820ae26d7dfeb0a7438f67557585735e4d601a74434620dfb6628a48a7112a25"
Oct 31 01:17:18.370645 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.370044    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"820ae26d7dfeb0a7438f67557585735e4d601a74434620dfb6628a48a7112a25"} err="failed to get container status \"820ae26d7dfeb0a7438f67557585735e4d601a74434620dfb6628a48a7112a25\": rpc error: code = NotFound desc = could not find container \"820ae26d7dfeb0a7438f67557585735e4d601a74434620dfb6628a48a7112a25\": container with ID starting with 820ae26d7dfeb0a7438f67557585735e4d601a74434620dfb6628a48a7112a25 not found: ID does not exist"
Oct 31 01:17:18.370645 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.370064    4327 scope.go:122] "RemoveContainer" containerID="ea13a63ccd5acd13266ebf8430f643cee616cd2a3cd83fb2e3a89e5b62a80297"
Oct 31 01:17:18.383306 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.383271    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfjg2\" (UniqueName: \"kubernetes.io/projected/5c54b47b-3540-45b9-8ac7-8b1f056b0b81-kube-api-access-nfjg2\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:17:18.383474 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.383312    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fmc6\" (UniqueName: \"kubernetes.io/projected/7900b962-7b7a-4d5e-b78f-8d8fe95d3123-kube-api-access-2fmc6\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:17:18.383474 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.383353    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c54b47b-3540-45b9-8ac7-8b1f056b0b81-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:17:18.383474 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.383368    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7900b962-7b7a-4d5e-b78f-8d8fe95d3123-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:17:18.387745 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.387719    4327 scope.go:122] "RemoveContainer" containerID="ea13a63ccd5acd13266ebf8430f643cee616cd2a3cd83fb2e3a89e5b62a80297"
Oct 31 01:17:18.388132 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:17:18.388111    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea13a63ccd5acd13266ebf8430f643cee616cd2a3cd83fb2e3a89e5b62a80297\": container with ID starting with ea13a63ccd5acd13266ebf8430f643cee616cd2a3cd83fb2e3a89e5b62a80297 not found: ID does not exist" containerID="ea13a63ccd5acd13266ebf8430f643cee616cd2a3cd83fb2e3a89e5b62a80297"
Oct 31 01:17:18.388234 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.388146    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea13a63ccd5acd13266ebf8430f643cee616cd2a3cd83fb2e3a89e5b62a80297"} err="failed to get container status \"ea13a63ccd5acd13266ebf8430f643cee616cd2a3cd83fb2e3a89e5b62a80297\": rpc error: code = NotFound desc = could not find container \"ea13a63ccd5acd13266ebf8430f643cee616cd2a3cd83fb2e3a89e5b62a80297\": container with ID starting with ea13a63ccd5acd13266ebf8430f643cee616cd2a3cd83fb2e3a89e5b62a80297 not found: ID does not exist"
Oct 31 01:17:18.388234 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.388170    4327 scope.go:122] "RemoveContainer" containerID="e3d33b14d748db76712767ff0171bbd9aeb76962dd305f51ccb35a9b074468aa"
Oct 31 01:17:18.401136 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.400867    4327 scope.go:122] "RemoveContainer" containerID="e3d33b14d748db76712767ff0171bbd9aeb76962dd305f51ccb35a9b074468aa"
Oct 31 01:17:18.401252 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:17:18.401202    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3d33b14d748db76712767ff0171bbd9aeb76962dd305f51ccb35a9b074468aa\": container with ID starting with e3d33b14d748db76712767ff0171bbd9aeb76962dd305f51ccb35a9b074468aa not found: ID does not exist" containerID="e3d33b14d748db76712767ff0171bbd9aeb76962dd305f51ccb35a9b074468aa"
Oct 31 01:17:18.401252 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.401233    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3d33b14d748db76712767ff0171bbd9aeb76962dd305f51ccb35a9b074468aa"} err="failed to get container status \"e3d33b14d748db76712767ff0171bbd9aeb76962dd305f51ccb35a9b074468aa\": rpc error: code = NotFound desc = could not find container \"e3d33b14d748db76712767ff0171bbd9aeb76962dd305f51ccb35a9b074468aa\": container with ID starting with e3d33b14d748db76712767ff0171bbd9aeb76962dd305f51ccb35a9b074468aa not found: ID does not exist"
Oct 31 01:17:18.405736 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.405709    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t"]
Oct 31 01:17:18.414234 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.414211    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e41918-7c4bf886f4-n572t"]
Oct 31 01:17:18.691174 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.691139    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h"]
Oct 31 01:17:18.698203 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.698175    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e41918-7dffbd4fbc-2ln6h"]
Oct 31 01:17:18.732732 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.732629    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57"]
Oct 31 01:17:18.739744 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:18.739713    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e41918-766c7f5686-8mf57"]
Oct 31 01:17:19.901257 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:19.901220    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d68696-884f5657-t256p"]
Oct 31 01:17:19.924156 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:19.924125    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c54b47b-3540-45b9-8ac7-8b1f056b0b81" path="/var/lib/kubelet/pods/5c54b47b-3540-45b9-8ac7-8b1f056b0b81/volumes"
Oct 31 01:17:19.924601 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:19.924588    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7900b962-7b7a-4d5e-b78f-8d8fe95d3123" path="/var/lib/kubelet/pods/7900b962-7b7a-4d5e-b78f-8d8fe95d3123/volumes"
Oct 31 01:17:19.925065 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:19.925049    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc74ef25-794f-4410-b3b7-403c7ba9f7d8" path="/var/lib/kubelet/pods/fc74ef25-794f-4410-b3b7-403c7ba9f7d8/volumes"
Oct 31 01:17:24.901777 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:24.901726    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-884f5657-t256p" podUID="165d92f7-faab-45a9-97e6-4f90ecd1cbd2" containerName="payload-processing" containerID="cri-o://ae12daa0248aa1fd2bfcaa6deb8990f138c3dd80b8ddadf2ec48f5b69d7a0605" gracePeriod=25
Oct 31 01:17:25.183887 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:25.183857    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-884f5657-t256p"
Oct 31 01:17:25.297009 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:25.296966    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/165d92f7-faab-45a9-97e6-4f90ecd1cbd2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/165d92f7-faab-45a9-97e6-4f90ecd1cbd2-plugins-config-volume\") pod \"165d92f7-faab-45a9-97e6-4f90ecd1cbd2\" (UID: \"165d92f7-faab-45a9-97e6-4f90ecd1cbd2\") "
Oct 31 01:17:25.297278 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:25.297261    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/165d92f7-faab-45a9-97e6-4f90ecd1cbd2-kube-api-access-8bpgt\" (UniqueName: \"kubernetes.io/projected/165d92f7-faab-45a9-97e6-4f90ecd1cbd2-kube-api-access-8bpgt\") pod \"165d92f7-faab-45a9-97e6-4f90ecd1cbd2\" (UID: \"165d92f7-faab-45a9-97e6-4f90ecd1cbd2\") "
Oct 31 01:17:25.297374 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:25.297350    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/165d92f7-faab-45a9-97e6-4f90ecd1cbd2-plugins-config-volume" pod "165d92f7-faab-45a9-97e6-4f90ecd1cbd2" (UID: "165d92f7-faab-45a9-97e6-4f90ecd1cbd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:17:25.297831 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:25.297806    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/165d92f7-faab-45a9-97e6-4f90ecd1cbd2-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:17:25.299572 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:25.299543    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165d92f7-faab-45a9-97e6-4f90ecd1cbd2-kube-api-access-8bpgt" pod "165d92f7-faab-45a9-97e6-4f90ecd1cbd2" (UID: "165d92f7-faab-45a9-97e6-4f90ecd1cbd2"). InnerVolumeSpecName "kube-api-access-8bpgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:17:25.399598 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:25.399564    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bpgt\" (UniqueName: \"kubernetes.io/projected/165d92f7-faab-45a9-97e6-4f90ecd1cbd2-kube-api-access-8bpgt\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:17:25.420357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:25.420268    4327 generic.go:356] "Generic (PLEG): container finished" podID="165d92f7-faab-45a9-97e6-4f90ecd1cbd2" containerID="ae12daa0248aa1fd2bfcaa6deb8990f138c3dd80b8ddadf2ec48f5b69d7a0605" exitCode=0
Oct 31 01:17:25.420357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:25.420309    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-884f5657-t256p"
Oct 31 01:17:25.420586 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:25.420357    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-884f5657-t256p" event={"ID":"165d92f7-faab-45a9-97e6-4f90ecd1cbd2","Type":"ContainerDied","Data":"ae12daa0248aa1fd2bfcaa6deb8990f138c3dd80b8ddadf2ec48f5b69d7a0605"}
Oct 31 01:17:25.420586 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:25.420406    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-884f5657-t256p" event={"ID":"165d92f7-faab-45a9-97e6-4f90ecd1cbd2","Type":"ContainerDied","Data":"6782004dc67fa07ddea91905920fdd1f96d7baef58c892a280404147141ffd66"}
Oct 31 01:17:25.420586 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:25.420426    4327 scope.go:122] "RemoveContainer" containerID="ae12daa0248aa1fd2bfcaa6deb8990f138c3dd80b8ddadf2ec48f5b69d7a0605"
Oct 31 01:17:25.431483 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:25.431458    4327 scope.go:122] "RemoveContainer" containerID="ae12daa0248aa1fd2bfcaa6deb8990f138c3dd80b8ddadf2ec48f5b69d7a0605"
Oct 31 01:17:25.431773 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:17:25.431750    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae12daa0248aa1fd2bfcaa6deb8990f138c3dd80b8ddadf2ec48f5b69d7a0605\": container with ID starting with ae12daa0248aa1fd2bfcaa6deb8990f138c3dd80b8ddadf2ec48f5b69d7a0605 not found: ID does not exist" containerID="ae12daa0248aa1fd2bfcaa6deb8990f138c3dd80b8ddadf2ec48f5b69d7a0605"
Oct 31 01:17:25.431871 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:25.431781    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae12daa0248aa1fd2bfcaa6deb8990f138c3dd80b8ddadf2ec48f5b69d7a0605"} err="failed to get container status \"ae12daa0248aa1fd2bfcaa6deb8990f138c3dd80b8ddadf2ec48f5b69d7a0605\": rpc error: code = NotFound desc = could not find container \"ae12daa0248aa1fd2bfcaa6deb8990f138c3dd80b8ddadf2ec48f5b69d7a0605\": container with ID starting with ae12daa0248aa1fd2bfcaa6deb8990f138c3dd80b8ddadf2ec48f5b69d7a0605 not found: ID does not exist"
Oct 31 01:17:25.475872 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:25.475842    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d68696-884f5657-t256p"]
Oct 31 01:17:25.479162 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:25.479141    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d68696-884f5657-t256p"]
Oct 31 01:17:25.924056 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:25.924025    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="165d92f7-faab-45a9-97e6-4f90ecd1cbd2" path="/var/lib/kubelet/pods/165d92f7-faab-45a9-97e6-4f90ecd1cbd2/volumes"
Oct 31 01:17:41.473220 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.473183    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk"]
Oct 31 01:17:41.473964 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.473948    4327 cpu_manager.go:414] "containerMap: removing container" podUID="5c54b47b-3540-45b9-8ac7-8b1f056b0b81" containerName="payload-pre-processing"
Oct 31 01:17:41.474022 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.473966    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c54b47b-3540-45b9-8ac7-8b1f056b0b81" containerName="payload-pre-processing"
Oct 31 01:17:41.474022 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.473983    4327 cpu_manager.go:414] "containerMap: removing container" podUID="7900b962-7b7a-4d5e-b78f-8d8fe95d3123" containerName="payload-processing"
Oct 31 01:17:41.474022 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.473988    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7900b962-7b7a-4d5e-b78f-8d8fe95d3123" containerName="payload-processing"
Oct 31 01:17:41.474022 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.474004    4327 cpu_manager.go:414] "containerMap: removing container" podUID="fc74ef25-794f-4410-b3b7-403c7ba9f7d8" containerName="maas-api"
Oct 31 01:17:41.474022 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.474009    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc74ef25-794f-4410-b3b7-403c7ba9f7d8" containerName="maas-api"
Oct 31 01:17:41.474022 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.474018    4327 cpu_manager.go:414] "containerMap: removing container" podUID="165d92f7-faab-45a9-97e6-4f90ecd1cbd2" containerName="payload-processing"
Oct 31 01:17:41.474022 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.474023    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="165d92f7-faab-45a9-97e6-4f90ecd1cbd2" containerName="payload-processing"
Oct 31 01:17:41.474288 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.474134    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c54b47b-3540-45b9-8ac7-8b1f056b0b81" containerName="payload-pre-processing"
Oct 31 01:17:41.474288 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.474145    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="7900b962-7b7a-4d5e-b78f-8d8fe95d3123" containerName="payload-processing"
Oct 31 01:17:41.474288 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.474153    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc74ef25-794f-4410-b3b7-403c7ba9f7d8" containerName="maas-api"
Oct 31 01:17:41.474288 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.474159    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="165d92f7-faab-45a9-97e6-4f90ecd1cbd2" containerName="payload-processing"
Oct 31 01:17:41.477955 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.477935    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk"
Oct 31 01:17:41.501746 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.498052    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-8d251f-serving-cert\""
Oct 31 01:17:41.503715 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.503331    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk"]
Oct 31 01:17:41.618828 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.618787    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6a22f7c-91e0-44af-bcf7-f987701bc859-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk\" (UID: \"b6a22f7c-91e0-44af-bcf7-f987701bc859\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk"
Oct 31 01:17:41.619025 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.618945    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6a22f7c-91e0-44af-bcf7-f987701bc859-metrics-certs\") pod \"maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk\" (UID: \"b6a22f7c-91e0-44af-bcf7-f987701bc859\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk"
Oct 31 01:17:41.619107 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.619073    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqd94\" (UniqueName: \"kubernetes.io/projected/b6a22f7c-91e0-44af-bcf7-f987701bc859-kube-api-access-tqd94\") pod \"maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk\" (UID: \"b6a22f7c-91e0-44af-bcf7-f987701bc859\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk"
Oct 31 01:17:41.619173 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.619146    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a22f7c-91e0-44af-bcf7-f987701bc859-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk\" (UID: \"b6a22f7c-91e0-44af-bcf7-f987701bc859\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk"
Oct 31 01:17:41.720292 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.720258    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a22f7c-91e0-44af-bcf7-f987701bc859-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk\" (UID: \"b6a22f7c-91e0-44af-bcf7-f987701bc859\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk"
Oct 31 01:17:41.720514 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.720355    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6a22f7c-91e0-44af-bcf7-f987701bc859-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk\" (UID: \"b6a22f7c-91e0-44af-bcf7-f987701bc859\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk"
Oct 31 01:17:41.720514 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.720446    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6a22f7c-91e0-44af-bcf7-f987701bc859-metrics-certs\") pod \"maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk\" (UID: \"b6a22f7c-91e0-44af-bcf7-f987701bc859\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk"
Oct 31 01:17:41.720651 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.720566    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqd94\" (UniqueName: \"kubernetes.io/projected/b6a22f7c-91e0-44af-bcf7-f987701bc859-kube-api-access-tqd94\") pod \"maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk\" (UID: \"b6a22f7c-91e0-44af-bcf7-f987701bc859\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk"
Oct 31 01:17:41.721109 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.721084    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a22f7c-91e0-44af-bcf7-f987701bc859-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk\" (UID: \"b6a22f7c-91e0-44af-bcf7-f987701bc859\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk"
Oct 31 01:17:41.722851 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.722827    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6a22f7c-91e0-44af-bcf7-f987701bc859-metrics-certs\") pod \"maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk\" (UID: \"b6a22f7c-91e0-44af-bcf7-f987701bc859\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk"
Oct 31 01:17:41.723033 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.723012    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6a22f7c-91e0-44af-bcf7-f987701bc859-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk\" (UID: \"b6a22f7c-91e0-44af-bcf7-f987701bc859\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk"
Oct 31 01:17:41.744947 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.744881    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqd94\" (UniqueName: \"kubernetes.io/projected/b6a22f7c-91e0-44af-bcf7-f987701bc859-kube-api-access-tqd94\") pod \"maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk\" (UID: \"b6a22f7c-91e0-44af-bcf7-f987701bc859\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk"
Oct 31 01:17:41.794181 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:41.794148    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk"
Oct 31 01:17:42.097188 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:17:42.097159    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a22f7c_91e0_44af_bcf7_f987701bc859.slice/crio-570cb20c27b9cbfefba5062f0b9fbd863feb5729a1ba080f7356256af36e1ac7 WatchSource:0}: Error finding container 570cb20c27b9cbfefba5062f0b9fbd863feb5729a1ba080f7356256af36e1ac7: Status 404 returned error can't find the container with id 570cb20c27b9cbfefba5062f0b9fbd863feb5729a1ba080f7356256af36e1ac7
Oct 31 01:17:42.098452 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:42.098426    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk"]
Oct 31 01:17:42.572383 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:42.572348    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk" event={"ID":"b6a22f7c-91e0-44af-bcf7-f987701bc859","Type":"ContainerStarted","Data":"570cb20c27b9cbfefba5062f0b9fbd863feb5729a1ba080f7356256af36e1ac7"}
Oct 31 01:17:43.582162 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:43.582120    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk" event={"ID":"b6a22f7c-91e0-44af-bcf7-f987701bc859","Type":"ContainerStarted","Data":"5c0cfbe8debebacd96c934ead2a56f0e020b07b04c96ba5699622b48c4328d24"}
Oct 31 01:17:43.582705 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:43.582344    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk"
Oct 31 01:17:43.627467 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:43.627416    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk" podStartSLOduration=2.099141892 podStartE2EDuration="2.627403182s" totalImagesPullingTime="528.26129ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:43.62561435 +0000 UTC m=+3523.847649893" watchObservedRunningTime="2026-10-31 01:17:43.627403182 +0000 UTC m=+3523.849438767"
Oct 31 01:17:45.320195 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:17:45.320130    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:17:49.227937 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:49.227901    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10893-ba2f72"]
Oct 31 01:17:49.232752 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:49.232731    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10893-ba2f72"
Oct 31 01:17:49.251340 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:49.251311    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10893-ba2f72"]
Oct 31 01:17:49.258915 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:49.258876    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8kc7d\""
Oct 31 01:17:49.353164 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:49.353120    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5ljb\" (UniqueName: \"kubernetes.io/projected/dc53e906-cc93-48b4-ba63-a73ab49b0ccc-kube-api-access-t5ljb\") pod \"test-curl-10893-ba2f72\" (UID: \"dc53e906-cc93-48b4-ba63-a73ab49b0ccc\") " pod="opendatahub/test-curl-10893-ba2f72"
Oct 31 01:17:49.454691 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:49.454644    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5ljb\" (UniqueName: \"kubernetes.io/projected/dc53e906-cc93-48b4-ba63-a73ab49b0ccc-kube-api-access-t5ljb\") pod \"test-curl-10893-ba2f72\" (UID: \"dc53e906-cc93-48b4-ba63-a73ab49b0ccc\") " pod="opendatahub/test-curl-10893-ba2f72"
Oct 31 01:17:49.478758 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:49.478654    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5ljb\" (UniqueName: \"kubernetes.io/projected/dc53e906-cc93-48b4-ba63-a73ab49b0ccc-kube-api-access-t5ljb\") pod \"test-curl-10893-ba2f72\" (UID: \"dc53e906-cc93-48b4-ba63-a73ab49b0ccc\") " pod="opendatahub/test-curl-10893-ba2f72"
Oct 31 01:17:49.546269 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:49.546230    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10893-ba2f72"
Oct 31 01:17:49.596640 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:49.596612    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk"
Oct 31 01:17:49.853386 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:17:49.853351    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc53e906_cc93_48b4_ba63_a73ab49b0ccc.slice/crio-620f6e3d0728a4af997e52adcece9819cb962de934bb071cdaa13f0b119f120d WatchSource:0}: Error finding container 620f6e3d0728a4af997e52adcece9819cb962de934bb071cdaa13f0b119f120d: Status 404 returned error can't find the container with id 620f6e3d0728a4af997e52adcece9819cb962de934bb071cdaa13f0b119f120d
Oct 31 01:17:49.854131 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:49.854106    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10893-ba2f72"]
Oct 31 01:17:50.645097 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:50.644913    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10893-ba2f72" event={"ID":"dc53e906-cc93-48b4-ba63-a73ab49b0ccc","Type":"ContainerStarted","Data":"620f6e3d0728a4af997e52adcece9819cb962de934bb071cdaa13f0b119f120d"}
Oct 31 01:17:53.670508 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:53.670465    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10893-ba2f72" event={"ID":"dc53e906-cc93-48b4-ba63-a73ab49b0ccc","Type":"ContainerStarted","Data":"d09a396c8189c0ecc860447e0771e19a1c35d9ecf22f629a296e792eef09e094"}
Oct 31 01:17:53.712435 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:53.712382    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-10893-ba2f72" podStartSLOduration=1.5234571510000001 podStartE2EDuration="4.712368173s" totalImagesPullingTime="3.188911022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:53.708944377 +0000 UTC m=+3533.930979922" watchObservedRunningTime="2026-10-31 01:17:53.712368173 +0000 UTC m=+3533.934403715"
Oct 31 01:17:54.626295 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:54.626260    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10893-ba2f72"]
Oct 31 01:17:54.632053 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:54.632023    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10893-ba2f72"]
Oct 31 01:17:54.787207 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:54.787169    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10893-0b39d5"]
Oct 31 01:17:54.787996 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:54.787978    4327 cpu_manager.go:414] "containerMap: removing container" podUID="dc53e906-cc93-48b4-ba63-a73ab49b0ccc" containerName="test-curl-10893-ba2f72"
Oct 31 01:17:54.788072 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:54.787997    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc53e906-cc93-48b4-ba63-a73ab49b0ccc" containerName="test-curl-10893-ba2f72"
Oct 31 01:17:54.788127 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:54.788118    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc53e906-cc93-48b4-ba63-a73ab49b0ccc" containerName="test-curl-10893-ba2f72"
Oct 31 01:17:54.792032 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:54.792007    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10893-0b39d5"
Oct 31 01:17:54.807005 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:54.806979    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10893-0b39d5"]
Oct 31 01:17:54.835295 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:54.835266    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wncs7\" (UniqueName: \"kubernetes.io/projected/99b98263-8ab9-4222-986b-23cf3c748479-kube-api-access-wncs7\") pod \"test-curl-10893-0b39d5\" (UID: \"99b98263-8ab9-4222-986b-23cf3c748479\") " pod="opendatahub/test-curl-10893-0b39d5"
Oct 31 01:17:54.941879 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:54.940867    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wncs7\" (UniqueName: \"kubernetes.io/projected/99b98263-8ab9-4222-986b-23cf3c748479-kube-api-access-wncs7\") pod \"test-curl-10893-0b39d5\" (UID: \"99b98263-8ab9-4222-986b-23cf3c748479\") " pod="opendatahub/test-curl-10893-0b39d5"
Oct 31 01:17:54.969273 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:54.969239    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wncs7\" (UniqueName: \"kubernetes.io/projected/99b98263-8ab9-4222-986b-23cf3c748479-kube-api-access-wncs7\") pod \"test-curl-10893-0b39d5\" (UID: \"99b98263-8ab9-4222-986b-23cf3c748479\") " pod="opendatahub/test-curl-10893-0b39d5"
Oct 31 01:17:55.105499 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:55.105455    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10893-0b39d5"
Oct 31 01:17:55.428438 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:17:55.428410    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99b98263_8ab9_4222_986b_23cf3c748479.slice/crio-5b4dfef64d3956effece571c66aeba6289fff0133e31060e0891a4599ab8e2e8 WatchSource:0}: Error finding container 5b4dfef64d3956effece571c66aeba6289fff0133e31060e0891a4599ab8e2e8: Status 404 returned error can't find the container with id 5b4dfef64d3956effece571c66aeba6289fff0133e31060e0891a4599ab8e2e8
Oct 31 01:17:55.431276 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:55.431252    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10893-0b39d5"]
Oct 31 01:17:55.688918 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:55.688821    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10893-0b39d5" event={"ID":"99b98263-8ab9-4222-986b-23cf3c748479","Type":"ContainerStarted","Data":"95c1995cb7a2dd78be78f17bbc5242f7aa2958c97272c2308a3458f04e10e9ea"}
Oct 31 01:17:55.688918 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:55.688872    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10893-0b39d5" event={"ID":"99b98263-8ab9-4222-986b-23cf3c748479","Type":"ContainerStarted","Data":"5b4dfef64d3956effece571c66aeba6289fff0133e31060e0891a4599ab8e2e8"}
Oct 31 01:17:55.689159 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:55.688955    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-10893-ba2f72" podUID="dc53e906-cc93-48b4-ba63-a73ab49b0ccc" containerName="test-curl-10893-ba2f72" containerID="cri-o://d09a396c8189c0ecc860447e0771e19a1c35d9ecf22f629a296e792eef09e094" gracePeriod=2
Oct 31 01:17:55.702912 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:17:55.702877    4327 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10893-ba2f72\" is forbidden: User \"system:node:ip-10-0-128-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-253.ec2.internal' and this object" podUID="dc53e906-cc93-48b4-ba63-a73ab49b0ccc" pod="opendatahub/test-curl-10893-ba2f72"
Oct 31 01:17:55.743605 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:55.743550    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-10893-0b39d5" podStartSLOduration=1.743537796 podStartE2EDuration="1.743537796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:55.74208048 +0000 UTC m=+3535.964116022" watchObservedRunningTime="2026-10-31 01:17:55.743537796 +0000 UTC m=+3535.965573338"
Oct 31 01:17:55.756080 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:17:55.756046    4327 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10893-ba2f72\" is forbidden: User \"system:node:ip-10-0-128-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-253.ec2.internal' and this object" podUID="dc53e906-cc93-48b4-ba63-a73ab49b0ccc" pod="opendatahub/test-curl-10893-ba2f72"
Oct 31 01:17:56.654353 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:56.654300    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd"]
Oct 31 01:17:56.661839 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:56.661813    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd"
Oct 31 01:17:56.676224 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:56.676195    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd"]
Oct 31 01:17:56.683739 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:56.683186    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d68696\""
Oct 31 01:17:56.683739 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:56.683296    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d68696-dockercfg-c878m\""
Oct 31 01:17:56.729050 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:56.728985    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10893-0b39d5"]
Oct 31 01:17:56.735417 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:56.735381    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10893-0b39d5"]
Oct 31 01:17:56.779180 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:56.779138    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e60feb74-e154-496d-88d1-9e8902b2e7ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd\" (UID: \"e60feb74-e154-496d-88d1-9e8902b2e7ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd"
Oct 31 01:17:56.779380 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:56.779201    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwc2m\" (UniqueName: \"kubernetes.io/projected/e60feb74-e154-496d-88d1-9e8902b2e7ae-kube-api-access-xwc2m\") pod \"payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd\" (UID: \"e60feb74-e154-496d-88d1-9e8902b2e7ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd"
Oct 31 01:17:56.832424 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:56.832386    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr"]
Oct 31 01:17:56.833218 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:56.833202    4327 cpu_manager.go:414] "containerMap: removing container" podUID="99b98263-8ab9-4222-986b-23cf3c748479" containerName="test-curl-10893-0b39d5"
Oct 31 01:17:56.833284 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:56.833219    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99b98263-8ab9-4222-986b-23cf3c748479" containerName="test-curl-10893-0b39d5"
Oct 31 01:17:56.833347 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:56.833337    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="99b98263-8ab9-4222-986b-23cf3c748479" containerName="test-curl-10893-0b39d5"
Oct 31 01:17:56.837093 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:56.837074    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr"
Oct 31 01:17:56.860569 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:56.860535    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr"]
Oct 31 01:17:56.880468 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:56.880420    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj65s\" (UniqueName: \"kubernetes.io/projected/fdeebb2d-1d8b-49e0-bdee-af282e78e7a1-kube-api-access-zj65s\") pod \"payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr\" (UID: \"fdeebb2d-1d8b-49e0-bdee-af282e78e7a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr"
Oct 31 01:17:56.880658 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:56.880618    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e60feb74-e154-496d-88d1-9e8902b2e7ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd\" (UID: \"e60feb74-e154-496d-88d1-9e8902b2e7ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd"
Oct 31 01:17:56.880786 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:56.880655    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwc2m\" (UniqueName: \"kubernetes.io/projected/e60feb74-e154-496d-88d1-9e8902b2e7ae-kube-api-access-xwc2m\") pod \"payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd\" (UID: \"e60feb74-e154-496d-88d1-9e8902b2e7ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd"
Oct 31 01:17:56.880878 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:56.880861    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdeebb2d-1d8b-49e0-bdee-af282e78e7a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr\" (UID: \"fdeebb2d-1d8b-49e0-bdee-af282e78e7a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr"
Oct 31 01:17:56.881614 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:56.881592    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e60feb74-e154-496d-88d1-9e8902b2e7ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd\" (UID: \"e60feb74-e154-496d-88d1-9e8902b2e7ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd"
Oct 31 01:17:56.904893 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:56.904825    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwc2m\" (UniqueName: \"kubernetes.io/projected/e60feb74-e154-496d-88d1-9e8902b2e7ae-kube-api-access-xwc2m\") pod \"payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd\" (UID: \"e60feb74-e154-496d-88d1-9e8902b2e7ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd"
Oct 31 01:17:56.982808 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:56.982765    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdeebb2d-1d8b-49e0-bdee-af282e78e7a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr\" (UID: \"fdeebb2d-1d8b-49e0-bdee-af282e78e7a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr"
Oct 31 01:17:56.983013 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:56.982870    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj65s\" (UniqueName: \"kubernetes.io/projected/fdeebb2d-1d8b-49e0-bdee-af282e78e7a1-kube-api-access-zj65s\") pod \"payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr\" (UID: \"fdeebb2d-1d8b-49e0-bdee-af282e78e7a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr"
Oct 31 01:17:56.983419 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:56.983398    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdeebb2d-1d8b-49e0-bdee-af282e78e7a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr\" (UID: \"fdeebb2d-1d8b-49e0-bdee-af282e78e7a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr"
Oct 31 01:17:56.984925 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:56.984906    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd"
Oct 31 01:17:57.007049 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:57.007025    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj65s\" (UniqueName: \"kubernetes.io/projected/fdeebb2d-1d8b-49e0-bdee-af282e78e7a1-kube-api-access-zj65s\") pod \"payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr\" (UID: \"fdeebb2d-1d8b-49e0-bdee-af282e78e7a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr"
Oct 31 01:17:57.151756 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:57.151721    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr"
Oct 31 01:17:57.295452 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:17:57.295424    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode60feb74_e154_496d_88d1_9e8902b2e7ae.slice/crio-8eb86ad0eef6b912a689fc565ead136fd62d47027760345a3fe5dd807a00da56 WatchSource:0}: Error finding container 8eb86ad0eef6b912a689fc565ead136fd62d47027760345a3fe5dd807a00da56: Status 404 returned error can't find the container with id 8eb86ad0eef6b912a689fc565ead136fd62d47027760345a3fe5dd807a00da56
Oct 31 01:17:57.298359 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:57.298334    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd"]
Oct 31 01:17:57.494353 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:17:57.494318    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdeebb2d_1d8b_49e0_bdee_af282e78e7a1.slice/crio-02c9087a6c492e47b6b52fa851c0ffcef0771c6b11cf976153e70d2959cddaaa WatchSource:0}: Error finding container 02c9087a6c492e47b6b52fa851c0ffcef0771c6b11cf976153e70d2959cddaaa: Status 404 returned error can't find the container with id 02c9087a6c492e47b6b52fa851c0ffcef0771c6b11cf976153e70d2959cddaaa
Oct 31 01:17:57.495175 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:57.495151    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr"]
Oct 31 01:17:57.716080 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:57.716047    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd" event={"ID":"e60feb74-e154-496d-88d1-9e8902b2e7ae","Type":"ContainerStarted","Data":"52d3a2d5f329673403c5f94f5fbb577785af0cb07ff7d84b54fab6cb3c29dc84"}
Oct 31 01:17:57.716538 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:57.716113    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd" event={"ID":"e60feb74-e154-496d-88d1-9e8902b2e7ae","Type":"ContainerStarted","Data":"8eb86ad0eef6b912a689fc565ead136fd62d47027760345a3fe5dd807a00da56"}
Oct 31 01:17:57.716538 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:57.716178    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd"
Oct 31 01:17:57.718022 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:57.717996    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr" event={"ID":"fdeebb2d-1d8b-49e0-bdee-af282e78e7a1","Type":"ContainerStarted","Data":"d1238ead261897c88a71f624d675cec283b8ebd3e15f78a4fee71b3978a69177"}
Oct 31 01:17:57.718154 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:57.718050    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr"
Oct 31 01:17:57.718154 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:57.718074    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr" event={"ID":"fdeebb2d-1d8b-49e0-bdee-af282e78e7a1","Type":"ContainerStarted","Data":"02c9087a6c492e47b6b52fa851c0ffcef0771c6b11cf976153e70d2959cddaaa"}
Oct 31 01:17:57.718265 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:57.718143    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-10893-0b39d5" podUID="99b98263-8ab9-4222-986b-23cf3c748479" containerName="test-curl-10893-0b39d5" containerID="cri-o://95c1995cb7a2dd78be78f17bbc5242f7aa2958c97272c2308a3458f04e10e9ea" gracePeriod=2
Oct 31 01:17:57.763017 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:57.762945    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd" podStartSLOduration=1.762918826 podStartE2EDuration="1.762918826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:57.757934178 +0000 UTC m=+3537.979969722" watchObservedRunningTime="2026-10-31 01:17:57.762918826 +0000 UTC m=+3537.984954370"
Oct 31 01:17:57.773479 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:17:57.772382    4327 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10893-0b39d5\" is forbidden: User \"system:node:ip-10-0-128-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-253.ec2.internal' and this object" podUID="99b98263-8ab9-4222-986b-23cf3c748479" pod="opendatahub/test-curl-10893-0b39d5"
Oct 31 01:17:57.819051 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:57.818905    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr" podStartSLOduration=1.818883095 podStartE2EDuration="1.818883095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:57.814172484 +0000 UTC m=+3538.036208025" watchObservedRunningTime="2026-10-31 01:17:57.818883095 +0000 UTC m=+3538.040918641"
Oct 31 01:17:57.888968 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:57.888944    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10893-ba2f72"
Oct 31 01:17:57.902885 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:17:57.902858    4327 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10893-0b39d5\" is forbidden: User \"system:node:ip-10-0-128-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-253.ec2.internal' and this object" podUID="99b98263-8ab9-4222-986b-23cf3c748479" pod="opendatahub/test-curl-10893-0b39d5"
Oct 31 01:17:57.996361 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:57.996260    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc53e906-cc93-48b4-ba63-a73ab49b0ccc-kube-api-access-t5ljb\" (UniqueName: \"kubernetes.io/projected/dc53e906-cc93-48b4-ba63-a73ab49b0ccc-kube-api-access-t5ljb\") pod \"dc53e906-cc93-48b4-ba63-a73ab49b0ccc\" (UID: \"dc53e906-cc93-48b4-ba63-a73ab49b0ccc\") "
Oct 31 01:17:57.998519 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:57.998491    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc53e906-cc93-48b4-ba63-a73ab49b0ccc-kube-api-access-t5ljb" pod "dc53e906-cc93-48b4-ba63-a73ab49b0ccc" (UID: "dc53e906-cc93-48b4-ba63-a73ab49b0ccc"). InnerVolumeSpecName "kube-api-access-t5ljb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:17:58.098669 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:58.098631    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5ljb\" (UniqueName: \"kubernetes.io/projected/dc53e906-cc93-48b4-ba63-a73ab49b0ccc-kube-api-access-t5ljb\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:17:58.728623 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:58.728584    4327 generic.go:356] "Generic (PLEG): container finished" podID="dc53e906-cc93-48b4-ba63-a73ab49b0ccc" containerID="d09a396c8189c0ecc860447e0771e19a1c35d9ecf22f629a296e792eef09e094" exitCode=137
Oct 31 01:17:58.729144 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:58.728626    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10893-ba2f72"
Oct 31 01:17:58.729144 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:58.728702    4327 scope.go:122] "RemoveContainer" containerID="d09a396c8189c0ecc860447e0771e19a1c35d9ecf22f629a296e792eef09e094"
Oct 31 01:17:58.740642 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:58.740617    4327 scope.go:122] "RemoveContainer" containerID="d09a396c8189c0ecc860447e0771e19a1c35d9ecf22f629a296e792eef09e094"
Oct 31 01:17:58.741004 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:17:58.740978    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d09a396c8189c0ecc860447e0771e19a1c35d9ecf22f629a296e792eef09e094\": container with ID starting with d09a396c8189c0ecc860447e0771e19a1c35d9ecf22f629a296e792eef09e094 not found: ID does not exist" containerID="d09a396c8189c0ecc860447e0771e19a1c35d9ecf22f629a296e792eef09e094"
Oct 31 01:17:58.741086 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:58.741016    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d09a396c8189c0ecc860447e0771e19a1c35d9ecf22f629a296e792eef09e094"} err="failed to get container status \"d09a396c8189c0ecc860447e0771e19a1c35d9ecf22f629a296e792eef09e094\": rpc error: code = NotFound desc = could not find container \"d09a396c8189c0ecc860447e0771e19a1c35d9ecf22f629a296e792eef09e094\": container with ID starting with d09a396c8189c0ecc860447e0771e19a1c35d9ecf22f629a296e792eef09e094 not found: ID does not exist"
Oct 31 01:17:59.891230 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:59.891169    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"]
Oct 31 01:17:59.902028 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:59.902003    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:17:59.919734 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:59.919536    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-36af31-openshift-default-dockercfg-vv85m\""
Oct 31 01:17:59.930488 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:59.930457    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc53e906-cc93-48b4-ba63-a73ab49b0ccc" path="/var/lib/kubelet/pods/dc53e906-cc93-48b4-ba63-a73ab49b0ccc/volumes"
Oct 31 01:17:59.946734 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:59.946199    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"]
Oct 31 01:17:59.962887 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:17:59.962857    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10893-0b39d5"
Oct 31 01:17:59.976853 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:17:59.976775    4327 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10893-0b39d5\" is forbidden: User \"system:node:ip-10-0-128-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-253.ec2.internal' and this object" podUID="99b98263-8ab9-4222-986b-23cf3c748479" pod="opendatahub/test-curl-10893-0b39d5"
Oct 31 01:18:00.029319 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.029285    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99b98263-8ab9-4222-986b-23cf3c748479-kube-api-access-wncs7\" (UniqueName: \"kubernetes.io/projected/99b98263-8ab9-4222-986b-23cf3c748479-kube-api-access-wncs7\") pod \"99b98263-8ab9-4222-986b-23cf3c748479\" (UID: \"99b98263-8ab9-4222-986b-23cf3c748479\") "
Oct 31 01:18:00.029585 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.029566    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-data\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.029652 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.029595    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-token\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.029652 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.029622    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istiod-ca-cert\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.030202 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.030176    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9snw8\" (UniqueName: \"kubernetes.io/projected/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-kube-api-access-9snw8\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.030339 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.030322    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-envoy\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.030399 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.030371    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-credential-socket\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.030558 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.030535    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-workload-socket\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.030618 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.030587    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-workload-certs\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.030618 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.030612    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-podinfo\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.031826 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.031795    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b98263-8ab9-4222-986b-23cf3c748479-kube-api-access-wncs7" pod "99b98263-8ab9-4222-986b-23cf3c748479" (UID: "99b98263-8ab9-4222-986b-23cf3c748479"). InnerVolumeSpecName "kube-api-access-wncs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:18:00.131606 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.131575    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-envoy\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.131803 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.131625    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-credential-socket\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.131803 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.131714    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-workload-socket\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.131803 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.131744    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-workload-certs\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.131803 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.131762    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-podinfo\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.131803 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.131795    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-data\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.132110 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.131816    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-token\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.132110 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.131844    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istiod-ca-cert\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.132110 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.131940    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9snw8\" (UniqueName: \"kubernetes.io/projected/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-kube-api-access-9snw8\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.132110 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.132036    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wncs7\" (UniqueName: \"kubernetes.io/projected/99b98263-8ab9-4222-986b-23cf3c748479-kube-api-access-wncs7\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:18:00.132262 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.132142    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-credential-socket\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.132631 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.132601    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-data\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.133603 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.133578    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istiod-ca-cert\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.133907 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.133886    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-workload-socket\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.134154 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.134133    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-workload-certs\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.134995 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.134968    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-envoy\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.138234 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.138207    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-podinfo\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.157947 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.157911    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9snw8\" (UniqueName: \"kubernetes.io/projected/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-kube-api-access-9snw8\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.158099 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.158025    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-token\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.272585 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.272511    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-36af31-openshift-default-dockercfg-vv85m\""
Oct 31 01:18:00.279553 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.279534    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:00.619238 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:18:00.619195    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe1cf54_bbbc_4c89_ba1a_06caf4bcd789.slice/crio-9720f736de1252c1124693b883538b02e180f50df57f8920449493139405bf31 WatchSource:0}: Error finding container 9720f736de1252c1124693b883538b02e180f50df57f8920449493139405bf31: Status 404 returned error can't find the container with id 9720f736de1252c1124693b883538b02e180f50df57f8920449493139405bf31
Oct 31 01:18:00.624261 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.624209    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:18:00.624420 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.624314    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:18:00.624420 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.624355    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:18:00.627794 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.627756    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"]
Oct 31 01:18:00.752126 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.752014    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd" event={"ID":"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789","Type":"ContainerStarted","Data":"ec34be9c4d862024b0bd3a17202ce963c416f9cf11fbd0b066a5fa48b2cedf9b"}
Oct 31 01:18:00.752126 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.752076    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd" event={"ID":"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789","Type":"ContainerStarted","Data":"9720f736de1252c1124693b883538b02e180f50df57f8920449493139405bf31"}
Oct 31 01:18:00.753619 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.753591    4327 generic.go:356] "Generic (PLEG): container finished" podID="99b98263-8ab9-4222-986b-23cf3c748479" containerID="95c1995cb7a2dd78be78f17bbc5242f7aa2958c97272c2308a3458f04e10e9ea" exitCode=137
Oct 31 01:18:00.753814 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.753632    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10893-0b39d5"
Oct 31 01:18:00.753814 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.753708    4327 scope.go:122] "RemoveContainer" containerID="95c1995cb7a2dd78be78f17bbc5242f7aa2958c97272c2308a3458f04e10e9ea"
Oct 31 01:18:00.767828 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.767649    4327 scope.go:122] "RemoveContainer" containerID="95c1995cb7a2dd78be78f17bbc5242f7aa2958c97272c2308a3458f04e10e9ea"
Oct 31 01:18:00.767995 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:18:00.767970    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c1995cb7a2dd78be78f17bbc5242f7aa2958c97272c2308a3458f04e10e9ea\": container with ID starting with 95c1995cb7a2dd78be78f17bbc5242f7aa2958c97272c2308a3458f04e10e9ea not found: ID does not exist" containerID="95c1995cb7a2dd78be78f17bbc5242f7aa2958c97272c2308a3458f04e10e9ea"
Oct 31 01:18:00.768053 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.768009    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c1995cb7a2dd78be78f17bbc5242f7aa2958c97272c2308a3458f04e10e9ea"} err="failed to get container status \"95c1995cb7a2dd78be78f17bbc5242f7aa2958c97272c2308a3458f04e10e9ea\": rpc error: code = NotFound desc = could not find container \"95c1995cb7a2dd78be78f17bbc5242f7aa2958c97272c2308a3458f04e10e9ea\": container with ID starting with 95c1995cb7a2dd78be78f17bbc5242f7aa2958c97272c2308a3458f04e10e9ea not found: ID does not exist"
Oct 31 01:18:00.803799 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:00.803707    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd" podStartSLOduration=1.803661706 podStartE2EDuration="1.803661706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:00.797412107 +0000 UTC m=+3541.019447652" watchObservedRunningTime="2026-10-31 01:18:00.803661706 +0000 UTC m=+3541.025697250"
Oct 31 01:18:00.810869 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:18:00.810824    4327 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10893-0b39d5\" is forbidden: User \"system:node:ip-10-0-128-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-253.ec2.internal' and this object" podUID="99b98263-8ab9-4222-986b-23cf3c748479" pod="opendatahub/test-curl-10893-0b39d5"
Oct 31 01:18:00.824694 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:18:00.824654    4327 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10893-0b39d5\" is forbidden: User \"system:node:ip-10-0-128-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-253.ec2.internal' and this object" podUID="99b98263-8ab9-4222-986b-23cf3c748479" pod="opendatahub/test-curl-10893-0b39d5"
Oct 31 01:18:01.280404 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:01.280372    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:01.282201 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:01.282175    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 31 01:18:01.282336 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:01.282222    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd" podUID="abe1cf54-bbbc-4c89-ba1a-06caf4bcd789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 31 01:18:01.953088 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:01.953038    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99b98263-8ab9-4222-986b-23cf3c748479" path="/var/lib/kubelet/pods/99b98263-8ab9-4222-986b-23cf3c748479/volumes"
Oct 31 01:18:02.280184 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:02.280106    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 31 01:18:02.280184 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:02.280158    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd" podUID="abe1cf54-bbbc-4c89-ba1a-06caf4bcd789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 31 01:18:03.280908 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:03.280871    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 31 01:18:03.281381 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:03.280933    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd" podUID="abe1cf54-bbbc-4c89-ba1a-06caf4bcd789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 31 01:18:04.280883 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:04.280852    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 31 01:18:04.281012 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:04.280905    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd" podUID="abe1cf54-bbbc-4c89-ba1a-06caf4bcd789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 31 01:18:05.280089 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:05.280049    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 31 01:18:05.280298 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:05.280113    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd" podUID="abe1cf54-bbbc-4c89-ba1a-06caf4bcd789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 31 01:18:06.285946 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:06.282061    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 31 01:18:06.285946 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:06.282138    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd" podUID="abe1cf54-bbbc-4c89-ba1a-06caf4bcd789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 31 01:18:07.285357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:07.285325    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:07.285808 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:07.285791    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:07.286425 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:07.286406    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:18:08.731241 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:08.731207    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd"
Oct 31 01:18:08.731653 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:08.731294    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr"
Oct 31 01:18:09.424938 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:09.424900    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt"]
Oct 31 01:18:09.425171 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:09.425142    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt" podUID="782150a7-2f48-4aac-9e16-8c3d6cb53a82" containerName="payload-pre-processing" containerID="cri-o://18959641a1a8b047f2e24030e0f6f2a4c4ad5ac220cf61549162458e4f8cfc83" gracePeriod=30
Oct 31 01:18:09.746386 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:09.746355    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt"
Oct 31 01:18:09.876191 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:09.876141    4327 generic.go:356] "Generic (PLEG): container finished" podID="782150a7-2f48-4aac-9e16-8c3d6cb53a82" containerID="18959641a1a8b047f2e24030e0f6f2a4c4ad5ac220cf61549162458e4f8cfc83" exitCode=0
Oct 31 01:18:09.876408 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:09.876354    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt" event={"ID":"782150a7-2f48-4aac-9e16-8c3d6cb53a82","Type":"ContainerDied","Data":"18959641a1a8b047f2e24030e0f6f2a4c4ad5ac220cf61549162458e4f8cfc83"}
Oct 31 01:18:09.876531 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:09.876516    4327 scope.go:122] "RemoveContainer" containerID="18959641a1a8b047f2e24030e0f6f2a4c4ad5ac220cf61549162458e4f8cfc83"
Oct 31 01:18:09.876790 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:09.876720    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt"
Oct 31 01:18:09.877110 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:09.876953    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt" event={"ID":"782150a7-2f48-4aac-9e16-8c3d6cb53a82","Type":"ContainerDied","Data":"a557c5e9b59d4c7c7653c52371db37b6508445e4b41875fd5aa2699dec05f9bb"}
Oct 31 01:18:09.898354 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:09.897826    4327 scope.go:122] "RemoveContainer" containerID="18959641a1a8b047f2e24030e0f6f2a4c4ad5ac220cf61549162458e4f8cfc83"
Oct 31 01:18:09.898354 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:18:09.898259    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18959641a1a8b047f2e24030e0f6f2a4c4ad5ac220cf61549162458e4f8cfc83\": container with ID starting with 18959641a1a8b047f2e24030e0f6f2a4c4ad5ac220cf61549162458e4f8cfc83 not found: ID does not exist" containerID="18959641a1a8b047f2e24030e0f6f2a4c4ad5ac220cf61549162458e4f8cfc83"
Oct 31 01:18:09.898354 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:09.898294    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18959641a1a8b047f2e24030e0f6f2a4c4ad5ac220cf61549162458e4f8cfc83"} err="failed to get container status \"18959641a1a8b047f2e24030e0f6f2a4c4ad5ac220cf61549162458e4f8cfc83\": rpc error: code = NotFound desc = could not find container \"18959641a1a8b047f2e24030e0f6f2a4c4ad5ac220cf61549162458e4f8cfc83\": container with ID starting with 18959641a1a8b047f2e24030e0f6f2a4c4ad5ac220cf61549162458e4f8cfc83 not found: ID does not exist"
Oct 31 01:18:09.906261 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:09.906235    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/782150a7-2f48-4aac-9e16-8c3d6cb53a82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/782150a7-2f48-4aac-9e16-8c3d6cb53a82-plugins-config-volume\") pod \"782150a7-2f48-4aac-9e16-8c3d6cb53a82\" (UID: \"782150a7-2f48-4aac-9e16-8c3d6cb53a82\") "
Oct 31 01:18:09.906421 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:09.906325    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/782150a7-2f48-4aac-9e16-8c3d6cb53a82-kube-api-access-bw9p8\" (UniqueName: \"kubernetes.io/projected/782150a7-2f48-4aac-9e16-8c3d6cb53a82-kube-api-access-bw9p8\") pod \"782150a7-2f48-4aac-9e16-8c3d6cb53a82\" (UID: \"782150a7-2f48-4aac-9e16-8c3d6cb53a82\") "
Oct 31 01:18:09.906697 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:09.906653    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/782150a7-2f48-4aac-9e16-8c3d6cb53a82-plugins-config-volume" pod "782150a7-2f48-4aac-9e16-8c3d6cb53a82" (UID: "782150a7-2f48-4aac-9e16-8c3d6cb53a82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:18:09.907029 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:09.907011    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/782150a7-2f48-4aac-9e16-8c3d6cb53a82-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:18:09.910138 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:09.910116    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/782150a7-2f48-4aac-9e16-8c3d6cb53a82-kube-api-access-bw9p8" pod "782150a7-2f48-4aac-9e16-8c3d6cb53a82" (UID: "782150a7-2f48-4aac-9e16-8c3d6cb53a82"). InnerVolumeSpecName "kube-api-access-bw9p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:18:10.008547 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:10.008433    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw9p8\" (UniqueName: \"kubernetes.io/projected/782150a7-2f48-4aac-9e16-8c3d6cb53a82-kube-api-access-bw9p8\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:18:10.229370 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:10.229332    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt"]
Oct 31 01:18:10.232754 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:10.232728    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-33d44f-5c9c676cc5-gzpqt"]
Oct 31 01:18:11.926599 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:11.926512    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="782150a7-2f48-4aac-9e16-8c3d6cb53a82" path="/var/lib/kubelet/pods/782150a7-2f48-4aac-9e16-8c3d6cb53a82/volumes"
Oct 31 01:18:12.334430 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.334345    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"]
Oct 31 01:18:12.335200 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.335181    4327 cpu_manager.go:414] "containerMap: removing container" podUID="782150a7-2f48-4aac-9e16-8c3d6cb53a82" containerName="payload-pre-processing"
Oct 31 01:18:12.335200 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.335200    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="782150a7-2f48-4aac-9e16-8c3d6cb53a82" containerName="payload-pre-processing"
Oct 31 01:18:12.335373 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.335331    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="782150a7-2f48-4aac-9e16-8c3d6cb53a82" containerName="payload-pre-processing"
Oct 31 01:18:12.340155 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.340135    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.373888 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.373817    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"]
Oct 31 01:18:12.438966 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.438928    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-data\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.439161 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.439036    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-envoy\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.439161 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.439107    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-credential-socket\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.439267 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.439167    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-workload-certs\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.439367 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.439343    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-podinfo\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.439452 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.439436    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istiod-ca-cert\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.439535 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.439516    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-workload-socket\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.439577 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.439555    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-token\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.439617 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.439583    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c92jd\" (UniqueName: \"kubernetes.io/projected/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-kube-api-access-c92jd\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.449282 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.449262    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-3c90d8-openshift-default-dockercfg-vpjjh\""
Oct 31 01:18:12.541257 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.541216    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-podinfo\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.541451 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.541311    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istiod-ca-cert\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.541451 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.541360    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-workload-socket\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.541451 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.541394    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-token\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.541451 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.541427    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c92jd\" (UniqueName: \"kubernetes.io/projected/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-kube-api-access-c92jd\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.541768 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.541537    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-data\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.541768 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.541575    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-envoy\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.541768 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.541605    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-credential-socket\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.541768 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.541700    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-workload-certs\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.542028 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.541867    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-workload-socket\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.542028 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.542014    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-data\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.542146 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.542110    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istiod-ca-cert\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.542203 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.542159    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-workload-certs\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.542387 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.542364    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-credential-socket\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.543946 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.543922    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-envoy\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.544085 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.544071    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-podinfo\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.565329 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.565294    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-token\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.565552 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.565532    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c92jd\" (UniqueName: \"kubernetes.io/projected/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-kube-api-access-c92jd\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.654657 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.654618    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:12.975501 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.975466    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"]
Oct 31 01:18:12.975990 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:18:12.975617    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d8d7b3f_f478_4b30_b9be_6b9b86ff1fbc.slice/crio-4a0ec045e984c1ed5cd7af8fe33c865e1e2599140eb9b58d77b9a6e2f0a90cfa WatchSource:0}: Error finding container 4a0ec045e984c1ed5cd7af8fe33c865e1e2599140eb9b58d77b9a6e2f0a90cfa: Status 404 returned error can't find the container with id 4a0ec045e984c1ed5cd7af8fe33c865e1e2599140eb9b58d77b9a6e2f0a90cfa
Oct 31 01:18:12.978736 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.978699    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:18:12.978823 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.978769    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:18:12.978823 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:12.978800    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:18:13.984961 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:13.984926    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz" event={"ID":"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc","Type":"ContainerStarted","Data":"aef30089f4d7c883445bcc72f9666b248762e41553a9aacb13680aa0eb94b0f9"}
Oct 31 01:18:13.985360 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:13.984980    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz" event={"ID":"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc","Type":"ContainerStarted","Data":"4a0ec045e984c1ed5cd7af8fe33c865e1e2599140eb9b58d77b9a6e2f0a90cfa"}
Oct 31 01:18:14.047755 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:14.042623    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz" podStartSLOduration=2.042604246 podStartE2EDuration="2.042604246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:14.040671083 +0000 UTC m=+3554.262706626" watchObservedRunningTime="2026-10-31 01:18:14.042604246 +0000 UTC m=+3554.264639789"
Oct 31 01:18:14.655176 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:14.655140    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:14.656640 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:14.656617    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 31 01:18:14.656765 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:14.656668    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz" podUID="5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 31 01:18:15.304940 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:18:15.304896    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:18:15.655706 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:15.655642    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 31 01:18:15.655907 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:15.655735    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz" podUID="5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 31 01:18:16.655823 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:16.655783    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 31 01:18:16.656274 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:16.655851    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz" podUID="5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 31 01:18:17.278016 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:17.277976    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"]
Oct 31 01:18:17.278274 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:17.278246    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2" podUID="cae467a1-6a9a-4c1b-855a-ac8edb1d10ee" containerName="istio-proxy" containerID="cri-o://edc647a438c1f9c18824aaf21ef6d30242b5a60a8b9d6ac65a20a68dc06649d6" gracePeriod=30
Oct 31 01:18:17.655055 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:17.655020    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 31 01:18:17.655246 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:17.655076    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz" podUID="5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 31 01:18:18.655738 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:18.655571    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 31 01:18:18.655738 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:18.655639    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz" podUID="5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 31 01:18:19.726430 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:19.726389    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:19.727211 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:19.727171    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:19.728813 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:19.728789    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:18:22.572594 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.572567    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:18:22.648166 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.648137    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-token\" (UniqueName: \"kubernetes.io/projected/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-token\") pod \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") "
Oct 31 01:18:22.648166 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.648173    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-workload-certs\") pod \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") "
Oct 31 01:18:22.648394 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.648289    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-kube-api-access-kdxmj\" (UniqueName: \"kubernetes.io/projected/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-kube-api-access-kdxmj\") pod \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") "
Oct 31 01:18:22.648394 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.648361    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-data\") pod \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") "
Oct 31 01:18:22.648506 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.648460    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-envoy\") pod \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") "
Oct 31 01:18:22.648572 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.648508    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-workload-socket\") pod \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") "
Oct 31 01:18:22.648572 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.648560    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-workload-certs" pod "cae467a1-6a9a-4c1b-855a-ac8edb1d10ee" (UID: "cae467a1-6a9a-4c1b-855a-ac8edb1d10ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:18:22.648709 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.648583    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-podinfo\") pod \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") "
Oct 31 01:18:22.648709 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.648637    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istiod-ca-cert\") pod \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") "
Oct 31 01:18:22.648827 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.648727    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-credential-socket\") pod \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\" (UID: \"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee\") "
Oct 31 01:18:22.648889 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.648865    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-workload-socket" pod "cae467a1-6a9a-4c1b-855a-ac8edb1d10ee" (UID: "cae467a1-6a9a-4c1b-855a-ac8edb1d10ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:18:22.649302 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.649164    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-credential-socket" pod "cae467a1-6a9a-4c1b-855a-ac8edb1d10ee" (UID: "cae467a1-6a9a-4c1b-855a-ac8edb1d10ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:18:22.649302 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.649272    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-data" pod "cae467a1-6a9a-4c1b-855a-ac8edb1d10ee" (UID: "cae467a1-6a9a-4c1b-855a-ac8edb1d10ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:18:22.649498 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.649428    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:18:22.649498 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.649447    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:18:22.649498 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.649465    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:18:22.649498 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.649480    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:18:22.649766 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.649552    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istiod-ca-cert" pod "cae467a1-6a9a-4c1b-855a-ac8edb1d10ee" (UID: "cae467a1-6a9a-4c1b-855a-ac8edb1d10ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:18:22.651165 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.651081    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-kube-api-access-kdxmj" pod "cae467a1-6a9a-4c1b-855a-ac8edb1d10ee" (UID: "cae467a1-6a9a-4c1b-855a-ac8edb1d10ee"). InnerVolumeSpecName "kube-api-access-kdxmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:18:22.651165 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.651111    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-token" pod "cae467a1-6a9a-4c1b-855a-ac8edb1d10ee" (UID: "cae467a1-6a9a-4c1b-855a-ac8edb1d10ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:18:22.651165 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.651163    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-envoy" pod "cae467a1-6a9a-4c1b-855a-ac8edb1d10ee" (UID: "cae467a1-6a9a-4c1b-855a-ac8edb1d10ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:18:22.651389 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.651304    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-podinfo" pod "cae467a1-6a9a-4c1b-855a-ac8edb1d10ee" (UID: "cae467a1-6a9a-4c1b-855a-ac8edb1d10ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:18:22.750513 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.750476    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:18:22.750513 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.750509    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdxmj\" (UniqueName: \"kubernetes.io/projected/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-kube-api-access-kdxmj\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:18:22.750513 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.750519    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:18:22.750806 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.750528    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:18:22.750806 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:22.750537    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:18:23.074449 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:23.074414    4327 generic.go:356] "Generic (PLEG): container finished" podID="cae467a1-6a9a-4c1b-855a-ac8edb1d10ee" containerID="edc647a438c1f9c18824aaf21ef6d30242b5a60a8b9d6ac65a20a68dc06649d6" exitCode=0
Oct 31 01:18:23.074663 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:23.074482    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"
Oct 31 01:18:23.074663 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:23.074493    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2" event={"ID":"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee","Type":"ContainerDied","Data":"edc647a438c1f9c18824aaf21ef6d30242b5a60a8b9d6ac65a20a68dc06649d6"}
Oct 31 01:18:23.074663 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:23.074549    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2" event={"ID":"cae467a1-6a9a-4c1b-855a-ac8edb1d10ee","Type":"ContainerDied","Data":"bbd8b494201e60b1c944c23a02a5f7bb1756353110fd0b80b7511797d223d9a3"}
Oct 31 01:18:23.074663 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:23.074575    4327 scope.go:122] "RemoveContainer" containerID="edc647a438c1f9c18824aaf21ef6d30242b5a60a8b9d6ac65a20a68dc06649d6"
Oct 31 01:18:23.137916 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:23.137884    4327 scope.go:122] "RemoveContainer" containerID="30ab02190365330996820b709bd42274b2e961ee11f70d487aa8a90bcb827acd"
Oct 31 01:18:23.149085 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:23.149066    4327 scope.go:122] "RemoveContainer" containerID="edc647a438c1f9c18824aaf21ef6d30242b5a60a8b9d6ac65a20a68dc06649d6"
Oct 31 01:18:23.149409 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:18:23.149387    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edc647a438c1f9c18824aaf21ef6d30242b5a60a8b9d6ac65a20a68dc06649d6\": container with ID starting with edc647a438c1f9c18824aaf21ef6d30242b5a60a8b9d6ac65a20a68dc06649d6 not found: ID does not exist" containerID="edc647a438c1f9c18824aaf21ef6d30242b5a60a8b9d6ac65a20a68dc06649d6"
Oct 31 01:18:23.149471 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:23.149417    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc647a438c1f9c18824aaf21ef6d30242b5a60a8b9d6ac65a20a68dc06649d6"} err="failed to get container status \"edc647a438c1f9c18824aaf21ef6d30242b5a60a8b9d6ac65a20a68dc06649d6\": rpc error: code = NotFound desc = could not find container \"edc647a438c1f9c18824aaf21ef6d30242b5a60a8b9d6ac65a20a68dc06649d6\": container with ID starting with edc647a438c1f9c18824aaf21ef6d30242b5a60a8b9d6ac65a20a68dc06649d6 not found: ID does not exist"
Oct 31 01:18:23.149471 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:23.149437    4327 scope.go:122] "RemoveContainer" containerID="30ab02190365330996820b709bd42274b2e961ee11f70d487aa8a90bcb827acd"
Oct 31 01:18:23.149735 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:18:23.149706    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30ab02190365330996820b709bd42274b2e961ee11f70d487aa8a90bcb827acd\": container with ID starting with 30ab02190365330996820b709bd42274b2e961ee11f70d487aa8a90bcb827acd not found: ID does not exist" containerID="30ab02190365330996820b709bd42274b2e961ee11f70d487aa8a90bcb827acd"
Oct 31 01:18:23.149787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:23.149737    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30ab02190365330996820b709bd42274b2e961ee11f70d487aa8a90bcb827acd"} err="failed to get container status \"30ab02190365330996820b709bd42274b2e961ee11f70d487aa8a90bcb827acd\": rpc error: code = NotFound desc = could not find container \"30ab02190365330996820b709bd42274b2e961ee11f70d487aa8a90bcb827acd\": container with ID starting with 30ab02190365330996820b709bd42274b2e961ee11f70d487aa8a90bcb827acd not found: ID does not exist"
Oct 31 01:18:23.166114 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:23.166088    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"]
Oct 31 01:18:23.169631 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:23.169609    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-33d44f-openshift-default-55d7454777-ckgx2"]
Oct 31 01:18:23.923758 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:23.923727    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cae467a1-6a9a-4c1b-855a-ac8edb1d10ee" path="/var/lib/kubelet/pods/cae467a1-6a9a-4c1b-855a-ac8edb1d10ee/volumes"
Oct 31 01:18:27.642817 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:27.642779    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7"]
Oct 31 01:18:28.605808 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:28.605772    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj"]
Oct 31 01:18:28.644646 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:28.644610    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl"]
Oct 31 01:18:32.643472 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:32.643429    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7" podUID="c1501512-ab3e-4907-880d-5b1f0bb68946" containerName="payload-processing" containerID="cri-o://b6fdc4da777ff56b5d0de8b544767b70c47a7712d788eedfb51087f48fe49368" gracePeriod=25
Oct 31 01:18:32.928410 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:32.928380    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7"
Oct 31 01:18:33.018738 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:33.018704    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1501512-ab3e-4907-880d-5b1f0bb68946-kube-api-access-k27tl\" (UniqueName: \"kubernetes.io/projected/c1501512-ab3e-4907-880d-5b1f0bb68946-kube-api-access-k27tl\") pod \"c1501512-ab3e-4907-880d-5b1f0bb68946\" (UID: \"c1501512-ab3e-4907-880d-5b1f0bb68946\") "
Oct 31 01:18:33.018956 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:33.018884    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1501512-ab3e-4907-880d-5b1f0bb68946-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1501512-ab3e-4907-880d-5b1f0bb68946-plugins-config-volume\") pod \"c1501512-ab3e-4907-880d-5b1f0bb68946\" (UID: \"c1501512-ab3e-4907-880d-5b1f0bb68946\") "
Oct 31 01:18:33.019503 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:33.019472    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1501512-ab3e-4907-880d-5b1f0bb68946-plugins-config-volume" pod "c1501512-ab3e-4907-880d-5b1f0bb68946" (UID: "c1501512-ab3e-4907-880d-5b1f0bb68946"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:18:33.021528 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:33.021494    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1501512-ab3e-4907-880d-5b1f0bb68946-kube-api-access-k27tl" pod "c1501512-ab3e-4907-880d-5b1f0bb68946" (UID: "c1501512-ab3e-4907-880d-5b1f0bb68946"). InnerVolumeSpecName "kube-api-access-k27tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:18:33.121568 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:33.121525    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1501512-ab3e-4907-880d-5b1f0bb68946-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:18:33.121568 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:33.121561    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k27tl\" (UniqueName: \"kubernetes.io/projected/c1501512-ab3e-4907-880d-5b1f0bb68946-kube-api-access-k27tl\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:18:33.194905 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:33.194818    4327 generic.go:356] "Generic (PLEG): container finished" podID="c1501512-ab3e-4907-880d-5b1f0bb68946" containerID="b6fdc4da777ff56b5d0de8b544767b70c47a7712d788eedfb51087f48fe49368" exitCode=0
Oct 31 01:18:33.195089 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:33.194910    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7"
Oct 31 01:18:33.195089 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:33.194938    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7" event={"ID":"c1501512-ab3e-4907-880d-5b1f0bb68946","Type":"ContainerDied","Data":"b6fdc4da777ff56b5d0de8b544767b70c47a7712d788eedfb51087f48fe49368"}
Oct 31 01:18:33.195089 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:33.195018    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7" event={"ID":"c1501512-ab3e-4907-880d-5b1f0bb68946","Type":"ContainerDied","Data":"528af64d82097dcdd10973a38f4d7a332424498b6e8543b1cda4966b61104871"}
Oct 31 01:18:33.195089 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:33.195049    4327 scope.go:122] "RemoveContainer" containerID="b6fdc4da777ff56b5d0de8b544767b70c47a7712d788eedfb51087f48fe49368"
Oct 31 01:18:33.210854 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:33.210821    4327 scope.go:122] "RemoveContainer" containerID="b6fdc4da777ff56b5d0de8b544767b70c47a7712d788eedfb51087f48fe49368"
Oct 31 01:18:33.211351 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:18:33.211323    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6fdc4da777ff56b5d0de8b544767b70c47a7712d788eedfb51087f48fe49368\": container with ID starting with b6fdc4da777ff56b5d0de8b544767b70c47a7712d788eedfb51087f48fe49368 not found: ID does not exist" containerID="b6fdc4da777ff56b5d0de8b544767b70c47a7712d788eedfb51087f48fe49368"
Oct 31 01:18:33.211430 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:33.211363    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6fdc4da777ff56b5d0de8b544767b70c47a7712d788eedfb51087f48fe49368"} err="failed to get container status \"b6fdc4da777ff56b5d0de8b544767b70c47a7712d788eedfb51087f48fe49368\": rpc error: code = NotFound desc = could not find container \"b6fdc4da777ff56b5d0de8b544767b70c47a7712d788eedfb51087f48fe49368\": container with ID starting with b6fdc4da777ff56b5d0de8b544767b70c47a7712d788eedfb51087f48fe49368 not found: ID does not exist"
Oct 31 01:18:33.250776 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:33.250746    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7"]
Oct 31 01:18:33.259965 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:33.259937    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8d251f-757d49dcf5-8lzp7"]
Oct 31 01:18:33.615107 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:33.611047    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj" podUID="f15fde05-d821-4a9f-bff6-d2ce68887b8b" containerName="payload-processing" containerID="cri-o://122d6c5a980a807f0da62a630b6d86c1124a3f07590131950e8b11fc996b1891" gracePeriod=25
Oct 31 01:18:33.645491 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:33.645447    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl" podUID="ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0" containerName="payload-pre-processing" containerID="cri-o://e50d732ba35f2960aa6409d0f8997b9e77528e93d72acd23e25c9d289d982add" gracePeriod=25
Oct 31 01:18:33.922875 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:33.922795    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1501512-ab3e-4907-880d-5b1f0bb68946" path="/var/lib/kubelet/pods/c1501512-ab3e-4907-880d-5b1f0bb68946/volumes"
Oct 31 01:18:41.189663 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:41.189621    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl" podUID="ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:18:41.190626 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:41.190594    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj" podUID="f15fde05-d821-4a9f-bff6-d2ce68887b8b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:18:45.294430 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:18:45.294388    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:18:51.176480 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:51.176435    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj" podUID="f15fde05-d821-4a9f-bff6-d2ce68887b8b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:18:51.176933 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:51.176442    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl" podUID="ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:18:53.807093 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:53.807068    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj"
Oct 31 01:18:53.929583 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:53.929557    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl"
Oct 31 01:18:53.936099 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:53.936075    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f15fde05-d821-4a9f-bff6-d2ce68887b8b-kube-api-access-r5575\" (UniqueName: \"kubernetes.io/projected/f15fde05-d821-4a9f-bff6-d2ce68887b8b-kube-api-access-r5575\") pod \"f15fde05-d821-4a9f-bff6-d2ce68887b8b\" (UID: \"f15fde05-d821-4a9f-bff6-d2ce68887b8b\") "
Oct 31 01:18:53.936234 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:53.936209    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f15fde05-d821-4a9f-bff6-d2ce68887b8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f15fde05-d821-4a9f-bff6-d2ce68887b8b-plugins-config-volume\") pod \"f15fde05-d821-4a9f-bff6-d2ce68887b8b\" (UID: \"f15fde05-d821-4a9f-bff6-d2ce68887b8b\") "
Oct 31 01:18:53.936560 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:53.936538    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f15fde05-d821-4a9f-bff6-d2ce68887b8b-plugins-config-volume" pod "f15fde05-d821-4a9f-bff6-d2ce68887b8b" (UID: "f15fde05-d821-4a9f-bff6-d2ce68887b8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:18:53.936746 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:53.936732    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f15fde05-d821-4a9f-bff6-d2ce68887b8b-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:18:53.938265 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:53.938239    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15fde05-d821-4a9f-bff6-d2ce68887b8b-kube-api-access-r5575" pod "f15fde05-d821-4a9f-bff6-d2ce68887b8b" (UID: "f15fde05-d821-4a9f-bff6-d2ce68887b8b"). InnerVolumeSpecName "kube-api-access-r5575". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:18:54.043236 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:54.043202    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0-kube-api-access-wnt2b\" (UniqueName: \"kubernetes.io/projected/ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0-kube-api-access-wnt2b\") pod \"ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0\" (UID: \"ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0\") "
Oct 31 01:18:54.043442 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:54.043423    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0-plugins-config-volume\") pod \"ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0\" (UID: \"ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0\") "
Oct 31 01:18:54.044504 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:54.044052    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0-plugins-config-volume" pod "ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0" (UID: "ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:18:54.044504 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:54.044427    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5575\" (UniqueName: \"kubernetes.io/projected/f15fde05-d821-4a9f-bff6-d2ce68887b8b-kube-api-access-r5575\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:18:54.044504 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:54.044451    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:18:54.046656 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:54.046633    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0-kube-api-access-wnt2b" pod "ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0" (UID: "ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0"). InnerVolumeSpecName "kube-api-access-wnt2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:18:54.145961 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:54.145924    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnt2b\" (UniqueName: \"kubernetes.io/projected/ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0-kube-api-access-wnt2b\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:18:54.435702 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:54.435651    4327 generic.go:356] "Generic (PLEG): container finished" podID="f15fde05-d821-4a9f-bff6-d2ce68887b8b" containerID="122d6c5a980a807f0da62a630b6d86c1124a3f07590131950e8b11fc996b1891" exitCode=0
Oct 31 01:18:54.436026 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:54.435767    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj"
Oct 31 01:18:54.436433 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:54.435790    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj" event={"ID":"f15fde05-d821-4a9f-bff6-d2ce68887b8b","Type":"ContainerDied","Data":"122d6c5a980a807f0da62a630b6d86c1124a3f07590131950e8b11fc996b1891"}
Oct 31 01:18:54.437174 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:54.437143    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj" event={"ID":"f15fde05-d821-4a9f-bff6-d2ce68887b8b","Type":"ContainerDied","Data":"1731aafc59211dcbe230e79ed69c8091f117b83a7b9b4376b785eea0320208f5"}
Oct 31 01:18:54.437518 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:54.437447    4327 scope.go:122] "RemoveContainer" containerID="122d6c5a980a807f0da62a630b6d86c1124a3f07590131950e8b11fc996b1891"
Oct 31 01:18:54.438027 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:54.438004    4327 generic.go:356] "Generic (PLEG): container finished" podID="ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0" containerID="e50d732ba35f2960aa6409d0f8997b9e77528e93d72acd23e25c9d289d982add" exitCode=0
Oct 31 01:18:54.438138 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:54.438074    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl" event={"ID":"ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0","Type":"ContainerDied","Data":"e50d732ba35f2960aa6409d0f8997b9e77528e93d72acd23e25c9d289d982add"}
Oct 31 01:18:54.438138 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:54.438109    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl" event={"ID":"ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0","Type":"ContainerDied","Data":"a07b0fe44a9fdb25fcc4570bf8edd524e957b0d807d9a20df9dce1520bb8012b"}
Oct 31 01:18:54.438265 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:54.438194    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl"
Oct 31 01:18:54.453322 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:54.453301    4327 scope.go:122] "RemoveContainer" containerID="122d6c5a980a807f0da62a630b6d86c1124a3f07590131950e8b11fc996b1891"
Oct 31 01:18:54.453753 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:18:54.453725    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"122d6c5a980a807f0da62a630b6d86c1124a3f07590131950e8b11fc996b1891\": container with ID starting with 122d6c5a980a807f0da62a630b6d86c1124a3f07590131950e8b11fc996b1891 not found: ID does not exist" containerID="122d6c5a980a807f0da62a630b6d86c1124a3f07590131950e8b11fc996b1891"
Oct 31 01:18:54.453893 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:54.453765    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"122d6c5a980a807f0da62a630b6d86c1124a3f07590131950e8b11fc996b1891"} err="failed to get container status \"122d6c5a980a807f0da62a630b6d86c1124a3f07590131950e8b11fc996b1891\": rpc error: code = NotFound desc = could not find container \"122d6c5a980a807f0da62a630b6d86c1124a3f07590131950e8b11fc996b1891\": container with ID starting with 122d6c5a980a807f0da62a630b6d86c1124a3f07590131950e8b11fc996b1891 not found: ID does not exist"
Oct 31 01:18:54.453893 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:54.453790    4327 scope.go:122] "RemoveContainer" containerID="e50d732ba35f2960aa6409d0f8997b9e77528e93d72acd23e25c9d289d982add"
Oct 31 01:18:54.465634 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:54.465603    4327 scope.go:122] "RemoveContainer" containerID="e50d732ba35f2960aa6409d0f8997b9e77528e93d72acd23e25c9d289d982add"
Oct 31 01:18:54.466054 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:18:54.466028    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e50d732ba35f2960aa6409d0f8997b9e77528e93d72acd23e25c9d289d982add\": container with ID starting with e50d732ba35f2960aa6409d0f8997b9e77528e93d72acd23e25c9d289d982add not found: ID does not exist" containerID="e50d732ba35f2960aa6409d0f8997b9e77528e93d72acd23e25c9d289d982add"
Oct 31 01:18:54.466182 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:54.466061    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50d732ba35f2960aa6409d0f8997b9e77528e93d72acd23e25c9d289d982add"} err="failed to get container status \"e50d732ba35f2960aa6409d0f8997b9e77528e93d72acd23e25c9d289d982add\": rpc error: code = NotFound desc = could not find container \"e50d732ba35f2960aa6409d0f8997b9e77528e93d72acd23e25c9d289d982add\": container with ID starting with e50d732ba35f2960aa6409d0f8997b9e77528e93d72acd23e25c9d289d982add not found: ID does not exist"
Oct 31 01:18:54.517546 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:54.517503    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj"]
Oct 31 01:18:54.521012 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:54.520984    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-476a67-6b894cfb5c-vljnj"]
Oct 31 01:18:54.555021 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:54.554987    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl"]
Oct 31 01:18:54.558263 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:54.558237    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-476a67-fcfc97bf9-6xmwl"]
Oct 31 01:18:55.923884 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:55.923844    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0" path="/var/lib/kubelet/pods/ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0/volumes"
Oct 31 01:18:55.924349 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:18:55.924334    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f15fde05-d821-4a9f-bff6-d2ce68887b8b" path="/var/lib/kubelet/pods/f15fde05-d821-4a9f-bff6-d2ce68887b8b/volumes"
Oct 31 01:19:07.760223 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:07.760185    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq"]
Oct 31 01:19:07.760763 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:07.760405    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq" podUID="273665c2-c5c0-4a57-930e-cfbd27ec4fe5" containerName="maas-api" containerID="cri-o://7073f13efcf956bdcc5148c94e28f37440b0e711bf1c50b34309cb9d81a6ac51" gracePeriod=30
Oct 31 01:19:08.144146 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:08.144108    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq"
Oct 31 01:19:08.326164 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:08.326120    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-kube-api-access-qpdx6\" (UniqueName: \"kubernetes.io/projected/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-kube-api-access-qpdx6\") pod \"273665c2-c5c0-4a57-930e-cfbd27ec4fe5\" (UID: \"273665c2-c5c0-4a57-930e-cfbd27ec4fe5\") "
Oct 31 01:19:08.326351 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:08.326202    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-metrics-certs\") pod \"273665c2-c5c0-4a57-930e-cfbd27ec4fe5\" (UID: \"273665c2-c5c0-4a57-930e-cfbd27ec4fe5\") "
Oct 31 01:19:08.326351 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:08.326277    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-maas-api-tls\") pod \"273665c2-c5c0-4a57-930e-cfbd27ec4fe5\" (UID: \"273665c2-c5c0-4a57-930e-cfbd27ec4fe5\") "
Oct 31 01:19:08.326487 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:08.326450    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-openshift-service-ca\") pod \"273665c2-c5c0-4a57-930e-cfbd27ec4fe5\" (UID: \"273665c2-c5c0-4a57-930e-cfbd27ec4fe5\") "
Oct 31 01:19:08.326912 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:08.326880    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-openshift-service-ca" pod "273665c2-c5c0-4a57-930e-cfbd27ec4fe5" (UID: "273665c2-c5c0-4a57-930e-cfbd27ec4fe5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:19:08.327722 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:08.327656    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:08.328931 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:08.328901    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-kube-api-access-qpdx6" pod "273665c2-c5c0-4a57-930e-cfbd27ec4fe5" (UID: "273665c2-c5c0-4a57-930e-cfbd27ec4fe5"). InnerVolumeSpecName "kube-api-access-qpdx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:19:08.329044 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:08.328954    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-metrics-certs" pod "273665c2-c5c0-4a57-930e-cfbd27ec4fe5" (UID: "273665c2-c5c0-4a57-930e-cfbd27ec4fe5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:19:08.329044 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:08.328979    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-maas-api-tls" pod "273665c2-c5c0-4a57-930e-cfbd27ec4fe5" (UID: "273665c2-c5c0-4a57-930e-cfbd27ec4fe5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:19:08.428551 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:08.428517    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpdx6\" (UniqueName: \"kubernetes.io/projected/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-kube-api-access-qpdx6\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:08.428551 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:08.428549    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:08.428551 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:08.428559    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/273665c2-c5c0-4a57-930e-cfbd27ec4fe5-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:08.544520 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:08.544486    4327 generic.go:356] "Generic (PLEG): container finished" podID="273665c2-c5c0-4a57-930e-cfbd27ec4fe5" containerID="7073f13efcf956bdcc5148c94e28f37440b0e711bf1c50b34309cb9d81a6ac51" exitCode=0
Oct 31 01:19:08.544736 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:08.544535    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq"
Oct 31 01:19:08.544736 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:08.544563    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq" event={"ID":"273665c2-c5c0-4a57-930e-cfbd27ec4fe5","Type":"ContainerDied","Data":"7073f13efcf956bdcc5148c94e28f37440b0e711bf1c50b34309cb9d81a6ac51"}
Oct 31 01:19:08.544736 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:08.544642    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq" event={"ID":"273665c2-c5c0-4a57-930e-cfbd27ec4fe5","Type":"ContainerDied","Data":"6fca16f2d36c4624f4939263cc2123a4e3acdf4b086cc113eba3f2a1b84922d7"}
Oct 31 01:19:08.544736 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:08.544665    4327 scope.go:122] "RemoveContainer" containerID="7073f13efcf956bdcc5148c94e28f37440b0e711bf1c50b34309cb9d81a6ac51"
Oct 31 01:19:08.556961 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:08.556942    4327 scope.go:122] "RemoveContainer" containerID="7073f13efcf956bdcc5148c94e28f37440b0e711bf1c50b34309cb9d81a6ac51"
Oct 31 01:19:08.557323 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:19:08.557302    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7073f13efcf956bdcc5148c94e28f37440b0e711bf1c50b34309cb9d81a6ac51\": container with ID starting with 7073f13efcf956bdcc5148c94e28f37440b0e711bf1c50b34309cb9d81a6ac51 not found: ID does not exist" containerID="7073f13efcf956bdcc5148c94e28f37440b0e711bf1c50b34309cb9d81a6ac51"
Oct 31 01:19:08.557411 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:08.557332    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7073f13efcf956bdcc5148c94e28f37440b0e711bf1c50b34309cb9d81a6ac51"} err="failed to get container status \"7073f13efcf956bdcc5148c94e28f37440b0e711bf1c50b34309cb9d81a6ac51\": rpc error: code = NotFound desc = could not find container \"7073f13efcf956bdcc5148c94e28f37440b0e711bf1c50b34309cb9d81a6ac51\": container with ID starting with 7073f13efcf956bdcc5148c94e28f37440b0e711bf1c50b34309cb9d81a6ac51 not found: ID does not exist"
Oct 31 01:19:08.602319 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:08.602266    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq"]
Oct 31 01:19:08.614218 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:08.614177    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-476a67-86f6c58f8-j9jsq"]
Oct 31 01:19:09.926339 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:09.926304    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="273665c2-c5c0-4a57-930e-cfbd27ec4fe5" path="/var/lib/kubelet/pods/273665c2-c5c0-4a57-930e-cfbd27ec4fe5/volumes"
Oct 31 01:19:12.490701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:12.489741    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c"]
Oct 31 01:19:12.547700 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:12.544604    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8"]
Oct 31 01:19:15.294806 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:19:15.294767    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:19:16.002612 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:16.002575    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"]
Oct 31 01:19:16.002928 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:16.002870    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz" podUID="13ff4194-5ef3-4f4d-b6e7-4cb5df757371" containerName="istio-proxy" containerID="cri-o://1b262e5ace803963b4ac324e64dbd0383d1eed6fba2b7dfe66bce7c67d7eddd1" gracePeriod=30
Oct 31 01:19:17.490505 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:17.490460    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c" podUID="5fe10361-91bc-4ff6-a6d4-faec973ad665" containerName="payload-processing" containerID="cri-o://62b842af61f45b8bdbd3df0a55ffc33bd6cf81d6390eb05636e8904a575db817" gracePeriod=25
Oct 31 01:19:17.545370 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:17.545326    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8" podUID="9949e8e5-8f11-4c06-8a7b-20e8640c2ac6" containerName="payload-pre-processing" containerID="cri-o://74253eb04a8953575096c0b5814519e61e35839ac1ee27fc56b57774965ddac1" gracePeriod=25
Oct 31 01:19:18.693612 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:18.693573    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:19:18.694060 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:18.693653    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz" podUID="13ff4194-5ef3-4f4d-b6e7-4cb5df757371" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:19:21.292191 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.292159    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:19:21.476474 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.476438    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-kube-api-access-j5lbw\" (UniqueName: \"kubernetes.io/projected/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-kube-api-access-j5lbw\") pod \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") "
Oct 31 01:19:21.476734 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.476498    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-token\" (UniqueName: \"kubernetes.io/projected/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-token\") pod \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") "
Oct 31 01:19:21.476734 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.476558    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-podinfo\") pod \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") "
Oct 31 01:19:21.476734 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.476693    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-credential-socket\") pod \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") "
Oct 31 01:19:21.476734 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.476729    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-workload-certs\") pod \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") "
Oct 31 01:19:21.477002 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.476796    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-envoy\") pod \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") "
Oct 31 01:19:21.477002 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.476912    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-data\") pod \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") "
Oct 31 01:19:21.477111 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.477049    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istiod-ca-cert\") pod \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") "
Oct 31 01:19:21.477164 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.477108    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-workload-socket\") pod \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\" (UID: \"13ff4194-5ef3-4f4d-b6e7-4cb5df757371\") "
Oct 31 01:19:21.477238 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.477210    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-data" pod "13ff4194-5ef3-4f4d-b6e7-4cb5df757371" (UID: "13ff4194-5ef3-4f4d-b6e7-4cb5df757371"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:19:21.477297 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.477020    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-credential-socket" pod "13ff4194-5ef3-4f4d-b6e7-4cb5df757371" (UID: "13ff4194-5ef3-4f4d-b6e7-4cb5df757371"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:19:21.477587 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.477567    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istiod-ca-cert" pod "13ff4194-5ef3-4f4d-b6e7-4cb5df757371" (UID: "13ff4194-5ef3-4f4d-b6e7-4cb5df757371"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:19:21.477824 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.477805    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:21.477875 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.477829    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:21.477875 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.477843    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:21.478117 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.478099    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-workload-socket" pod "13ff4194-5ef3-4f4d-b6e7-4cb5df757371" (UID: "13ff4194-5ef3-4f4d-b6e7-4cb5df757371"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:19:21.478172 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.478156    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-workload-certs" pod "13ff4194-5ef3-4f4d-b6e7-4cb5df757371" (UID: "13ff4194-5ef3-4f4d-b6e7-4cb5df757371"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:19:21.478938 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.478911    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-kube-api-access-j5lbw" pod "13ff4194-5ef3-4f4d-b6e7-4cb5df757371" (UID: "13ff4194-5ef3-4f4d-b6e7-4cb5df757371"). InnerVolumeSpecName "kube-api-access-j5lbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:19:21.479121 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.479102    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-podinfo" pod "13ff4194-5ef3-4f4d-b6e7-4cb5df757371" (UID: "13ff4194-5ef3-4f4d-b6e7-4cb5df757371"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:19:21.479201 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.479180    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-token" pod "13ff4194-5ef3-4f4d-b6e7-4cb5df757371" (UID: "13ff4194-5ef3-4f4d-b6e7-4cb5df757371"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:19:21.479915 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.479889    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-envoy" pod "13ff4194-5ef3-4f4d-b6e7-4cb5df757371" (UID: "13ff4194-5ef3-4f4d-b6e7-4cb5df757371"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:19:21.579114 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.579030    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:21.579114 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.579061    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:21.579114 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.579071    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5lbw\" (UniqueName: \"kubernetes.io/projected/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-kube-api-access-j5lbw\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:21.579114 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.579081    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:21.579114 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.579090    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:21.579114 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.579097    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13ff4194-5ef3-4f4d-b6e7-4cb5df757371-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:21.656296 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.656266    4327 generic.go:356] "Generic (PLEG): container finished" podID="13ff4194-5ef3-4f4d-b6e7-4cb5df757371" containerID="1b262e5ace803963b4ac324e64dbd0383d1eed6fba2b7dfe66bce7c67d7eddd1" exitCode=0
Oct 31 01:19:21.656497 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.656333    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"
Oct 31 01:19:21.656497 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.656362    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz" event={"ID":"13ff4194-5ef3-4f4d-b6e7-4cb5df757371","Type":"ContainerDied","Data":"1b262e5ace803963b4ac324e64dbd0383d1eed6fba2b7dfe66bce7c67d7eddd1"}
Oct 31 01:19:21.656497 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.656421    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz" event={"ID":"13ff4194-5ef3-4f4d-b6e7-4cb5df757371","Type":"ContainerDied","Data":"33d78ee06f6b194d45b125df8ad485b09427a80b71517e2dbd9cc04bb39e106b"}
Oct 31 01:19:21.656497 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.656446    4327 scope.go:122] "RemoveContainer" containerID="1b262e5ace803963b4ac324e64dbd0383d1eed6fba2b7dfe66bce7c67d7eddd1"
Oct 31 01:19:21.667804 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.667787    4327 scope.go:122] "RemoveContainer" containerID="fb69bc630de7bea5dd4698a8804940d4e0b4314d8bba5a78937eabfc882e2f05"
Oct 31 01:19:21.678885 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.678863    4327 scope.go:122] "RemoveContainer" containerID="1b262e5ace803963b4ac324e64dbd0383d1eed6fba2b7dfe66bce7c67d7eddd1"
Oct 31 01:19:21.679210 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:19:21.679180    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b262e5ace803963b4ac324e64dbd0383d1eed6fba2b7dfe66bce7c67d7eddd1\": container with ID starting with 1b262e5ace803963b4ac324e64dbd0383d1eed6fba2b7dfe66bce7c67d7eddd1 not found: ID does not exist" containerID="1b262e5ace803963b4ac324e64dbd0383d1eed6fba2b7dfe66bce7c67d7eddd1"
Oct 31 01:19:21.679261 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.679223    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b262e5ace803963b4ac324e64dbd0383d1eed6fba2b7dfe66bce7c67d7eddd1"} err="failed to get container status \"1b262e5ace803963b4ac324e64dbd0383d1eed6fba2b7dfe66bce7c67d7eddd1\": rpc error: code = NotFound desc = could not find container \"1b262e5ace803963b4ac324e64dbd0383d1eed6fba2b7dfe66bce7c67d7eddd1\": container with ID starting with 1b262e5ace803963b4ac324e64dbd0383d1eed6fba2b7dfe66bce7c67d7eddd1 not found: ID does not exist"
Oct 31 01:19:21.679261 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.679242    4327 scope.go:122] "RemoveContainer" containerID="fb69bc630de7bea5dd4698a8804940d4e0b4314d8bba5a78937eabfc882e2f05"
Oct 31 01:19:21.679546 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:19:21.679519    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb69bc630de7bea5dd4698a8804940d4e0b4314d8bba5a78937eabfc882e2f05\": container with ID starting with fb69bc630de7bea5dd4698a8804940d4e0b4314d8bba5a78937eabfc882e2f05 not found: ID does not exist" containerID="fb69bc630de7bea5dd4698a8804940d4e0b4314d8bba5a78937eabfc882e2f05"
Oct 31 01:19:21.679604 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.679553    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb69bc630de7bea5dd4698a8804940d4e0b4314d8bba5a78937eabfc882e2f05"} err="failed to get container status \"fb69bc630de7bea5dd4698a8804940d4e0b4314d8bba5a78937eabfc882e2f05\": rpc error: code = NotFound desc = could not find container \"fb69bc630de7bea5dd4698a8804940d4e0b4314d8bba5a78937eabfc882e2f05\": container with ID starting with fb69bc630de7bea5dd4698a8804940d4e0b4314d8bba5a78937eabfc882e2f05 not found: ID does not exist"
Oct 31 01:19:21.711294 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.711261    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"]
Oct 31 01:19:21.723171 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.723142    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-476a67-openshift-default-9bfcb45d7-ckgxz"]
Oct 31 01:19:21.923210 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:21.923178    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13ff4194-5ef3-4f4d-b6e7-4cb5df757371" path="/var/lib/kubelet/pods/13ff4194-5ef3-4f4d-b6e7-4cb5df757371/volumes"
Oct 31 01:19:26.577483 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:26.577447    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8" podUID="9949e8e5-8f11-4c06-8a7b-20e8640c2ac6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:19:26.577938 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:26.577444    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c" podUID="5fe10361-91bc-4ff6-a6d4-faec973ad665" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:19:27.051387 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:27.051349    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97"]
Oct 31 01:19:27.082718 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:27.082669    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t"]
Oct 31 01:19:31.644297 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.644259    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n"]
Oct 31 01:19:31.645032 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.645017    4327 cpu_manager.go:414] "containerMap: removing container" podUID="cae467a1-6a9a-4c1b-855a-ac8edb1d10ee" containerName="istio-proxy"
Oct 31 01:19:31.645106 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.645033    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cae467a1-6a9a-4c1b-855a-ac8edb1d10ee" containerName="istio-proxy"
Oct 31 01:19:31.645106 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.645049    4327 cpu_manager.go:414] "containerMap: removing container" podUID="ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0" containerName="payload-pre-processing"
Oct 31 01:19:31.645106 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.645055    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0" containerName="payload-pre-processing"
Oct 31 01:19:31.645106 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.645065    4327 cpu_manager.go:414] "containerMap: removing container" podUID="c1501512-ab3e-4907-880d-5b1f0bb68946" containerName="payload-processing"
Oct 31 01:19:31.645106 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.645070    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1501512-ab3e-4907-880d-5b1f0bb68946" containerName="payload-processing"
Oct 31 01:19:31.645106 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.645082    4327 cpu_manager.go:414] "containerMap: removing container" podUID="f15fde05-d821-4a9f-bff6-d2ce68887b8b" containerName="payload-processing"
Oct 31 01:19:31.645106 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.645087    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f15fde05-d821-4a9f-bff6-d2ce68887b8b" containerName="payload-processing"
Oct 31 01:19:31.645106 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.645098    4327 cpu_manager.go:414] "containerMap: removing container" podUID="13ff4194-5ef3-4f4d-b6e7-4cb5df757371" containerName="istio-proxy"
Oct 31 01:19:31.645106 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.645104    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ff4194-5ef3-4f4d-b6e7-4cb5df757371" containerName="istio-proxy"
Oct 31 01:19:31.645394 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.645119    4327 cpu_manager.go:414] "containerMap: removing container" podUID="273665c2-c5c0-4a57-930e-cfbd27ec4fe5" containerName="maas-api"
Oct 31 01:19:31.645394 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.645123    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="273665c2-c5c0-4a57-930e-cfbd27ec4fe5" containerName="maas-api"
Oct 31 01:19:31.645394 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.645144    4327 cpu_manager.go:414] "containerMap: removing container" podUID="13ff4194-5ef3-4f4d-b6e7-4cb5df757371" containerName="istio-proxy"
Oct 31 01:19:31.645394 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.645150    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ff4194-5ef3-4f4d-b6e7-4cb5df757371" containerName="istio-proxy"
Oct 31 01:19:31.645394 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.645232    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="f15fde05-d821-4a9f-bff6-d2ce68887b8b" containerName="payload-processing"
Oct 31 01:19:31.645394 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.645243    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="13ff4194-5ef3-4f4d-b6e7-4cb5df757371" containerName="istio-proxy"
Oct 31 01:19:31.645394 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.645252    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="273665c2-c5c0-4a57-930e-cfbd27ec4fe5" containerName="maas-api"
Oct 31 01:19:31.645394 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.645260    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="cae467a1-6a9a-4c1b-855a-ac8edb1d10ee" containerName="istio-proxy"
Oct 31 01:19:31.645394 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.645269    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="cae467a1-6a9a-4c1b-855a-ac8edb1d10ee" containerName="istio-proxy"
Oct 31 01:19:31.645394 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.645275    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebc9ebd5-e3d6-48ed-9a8e-c11e7ef200e0" containerName="payload-pre-processing"
Oct 31 01:19:31.645394 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.645280    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1501512-ab3e-4907-880d-5b1f0bb68946" containerName="payload-processing"
Oct 31 01:19:31.649138 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.649119    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n"
Oct 31 01:19:31.671927 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.671898    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-114c95-serving-cert\""
Oct 31 01:19:31.676962 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.676929    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n"]
Oct 31 01:19:31.713317 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.713277    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-metrics-certs\") pod \"maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n\" (UID: \"ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n"
Oct 31 01:19:31.713522 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.713439    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n\" (UID: \"ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n"
Oct 31 01:19:31.713792 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.713769    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gscks\" (UniqueName: \"kubernetes.io/projected/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-kube-api-access-gscks\") pod \"maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n\" (UID: \"ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n"
Oct 31 01:19:31.713990 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.713813    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-maas-api-tls\") pod \"maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n\" (UID: \"ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n"
Oct 31 01:19:31.815164 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.815128    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-metrics-certs\") pod \"maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n\" (UID: \"ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n"
Oct 31 01:19:31.815385 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.815203    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n\" (UID: \"ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n"
Oct 31 01:19:31.815385 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.815334    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gscks\" (UniqueName: \"kubernetes.io/projected/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-kube-api-access-gscks\") pod \"maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n\" (UID: \"ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n"
Oct 31 01:19:31.815385 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.815359    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-maas-api-tls\") pod \"maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n\" (UID: \"ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n"
Oct 31 01:19:31.816043 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.816014    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n\" (UID: \"ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n"
Oct 31 01:19:31.817832 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.817805    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-metrics-certs\") pod \"maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n\" (UID: \"ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n"
Oct 31 01:19:31.817832 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.817824    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-maas-api-tls\") pod \"maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n\" (UID: \"ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n"
Oct 31 01:19:31.839381 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.839349    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gscks\" (UniqueName: \"kubernetes.io/projected/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-kube-api-access-gscks\") pod \"maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n\" (UID: \"ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n"
Oct 31 01:19:31.965245 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:31.965159    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n"
Oct 31 01:19:32.052168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:32.051982    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97" podUID="96e41c65-75e1-4501-b965-14fa687c68c9" containerName="payload-processing" containerID="cri-o://123afc8e585a785406de80a2f2a1de1cbb1f8b79e8f3508795ee8e4a5a303904" gracePeriod=25
Oct 31 01:19:32.083114 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:32.083066    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t" podUID="852c332c-c02b-4ccc-b4ef-8733645922c9" containerName="payload-pre-processing" containerID="cri-o://ad07de1e5863ffa3fa213ca07b65e6868c2974aa5977ff4dab513c593e426427" gracePeriod=25
Oct 31 01:19:32.288573 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:32.288532    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n"]
Oct 31 01:19:32.290016 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:32.289987    4327 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:19:32.789703 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:32.785958    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n" event={"ID":"ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf","Type":"ContainerStarted","Data":"0d4a078bb7492abb7be0be6745d8429f4dc1791147b2981a9b86ca6447e642e1"}
Oct 31 01:19:33.793859 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:33.793821    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n" event={"ID":"ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf","Type":"ContainerStarted","Data":"64d1de9a8acedbd1fdbedc370885c798ae7ee7fdcab3f106816cfbebc11b52f2"}
Oct 31 01:19:33.794285 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:33.793983    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n"
Oct 31 01:19:33.845633 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:33.845576    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n" podStartSLOduration=2.306561522 podStartE2EDuration="2.845562367s" totalImagesPullingTime="539.000845ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:33.842749116 +0000 UTC m=+3634.064784657" watchObservedRunningTime="2026-10-31 01:19:33.845562367 +0000 UTC m=+3634.067597909"
Oct 31 01:19:36.577862 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:36.577823    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c" podUID="5fe10361-91bc-4ff6-a6d4-faec973ad665" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:19:36.578275 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:36.577835    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8" podUID="9949e8e5-8f11-4c06-8a7b-20e8640c2ac6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:19:37.820584 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:37.819087    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c"
Oct 31 01:19:37.838329 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:37.838291    4327 generic.go:356] "Generic (PLEG): container finished" podID="9949e8e5-8f11-4c06-8a7b-20e8640c2ac6" containerID="74253eb04a8953575096c0b5814519e61e35839ac1ee27fc56b57774965ddac1" exitCode=0
Oct 31 01:19:37.838499 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:37.838369    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8" event={"ID":"9949e8e5-8f11-4c06-8a7b-20e8640c2ac6","Type":"ContainerDied","Data":"74253eb04a8953575096c0b5814519e61e35839ac1ee27fc56b57774965ddac1"}
Oct 31 01:19:37.840185 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:37.840150    4327 generic.go:356] "Generic (PLEG): container finished" podID="5fe10361-91bc-4ff6-a6d4-faec973ad665" containerID="62b842af61f45b8bdbd3df0a55ffc33bd6cf81d6390eb05636e8904a575db817" exitCode=0
Oct 31 01:19:37.840322 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:37.840221    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c"
Oct 31 01:19:37.840322 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:37.840228    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c" event={"ID":"5fe10361-91bc-4ff6-a6d4-faec973ad665","Type":"ContainerDied","Data":"62b842af61f45b8bdbd3df0a55ffc33bd6cf81d6390eb05636e8904a575db817"}
Oct 31 01:19:37.840322 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:37.840278    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c" event={"ID":"5fe10361-91bc-4ff6-a6d4-faec973ad665","Type":"ContainerDied","Data":"11c2bebd9e9fc900f3695b62eb103594eadde2b7978f29a7d666970c04a5192c"}
Oct 31 01:19:37.840322 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:37.840302    4327 scope.go:122] "RemoveContainer" containerID="62b842af61f45b8bdbd3df0a55ffc33bd6cf81d6390eb05636e8904a575db817"
Oct 31 01:19:37.851058 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:37.851034    4327 scope.go:122] "RemoveContainer" containerID="62b842af61f45b8bdbd3df0a55ffc33bd6cf81d6390eb05636e8904a575db817"
Oct 31 01:19:37.851374 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:19:37.851347    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62b842af61f45b8bdbd3df0a55ffc33bd6cf81d6390eb05636e8904a575db817\": container with ID starting with 62b842af61f45b8bdbd3df0a55ffc33bd6cf81d6390eb05636e8904a575db817 not found: ID does not exist" containerID="62b842af61f45b8bdbd3df0a55ffc33bd6cf81d6390eb05636e8904a575db817"
Oct 31 01:19:37.851472 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:37.851382    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b842af61f45b8bdbd3df0a55ffc33bd6cf81d6390eb05636e8904a575db817"} err="failed to get container status \"62b842af61f45b8bdbd3df0a55ffc33bd6cf81d6390eb05636e8904a575db817\": rpc error: code = NotFound desc = could not find container \"62b842af61f45b8bdbd3df0a55ffc33bd6cf81d6390eb05636e8904a575db817\": container with ID starting with 62b842af61f45b8bdbd3df0a55ffc33bd6cf81d6390eb05636e8904a575db817 not found: ID does not exist"
Oct 31 01:19:37.881015 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:37.880985    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8"
Oct 31 01:19:37.905127 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:37.905095    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fe10361-91bc-4ff6-a6d4-faec973ad665-kube-api-access-tbh4v\" (UniqueName: \"kubernetes.io/projected/5fe10361-91bc-4ff6-a6d4-faec973ad665-kube-api-access-tbh4v\") pod \"5fe10361-91bc-4ff6-a6d4-faec973ad665\" (UID: \"5fe10361-91bc-4ff6-a6d4-faec973ad665\") "
Oct 31 01:19:37.905327 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:37.905168    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fe10361-91bc-4ff6-a6d4-faec973ad665-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fe10361-91bc-4ff6-a6d4-faec973ad665-plugins-config-volume\") pod \"5fe10361-91bc-4ff6-a6d4-faec973ad665\" (UID: \"5fe10361-91bc-4ff6-a6d4-faec973ad665\") "
Oct 31 01:19:37.905666 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:37.905633    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe10361-91bc-4ff6-a6d4-faec973ad665-plugins-config-volume" pod "5fe10361-91bc-4ff6-a6d4-faec973ad665" (UID: "5fe10361-91bc-4ff6-a6d4-faec973ad665"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:19:37.905821 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:37.905787    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fe10361-91bc-4ff6-a6d4-faec973ad665-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:37.907583 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:37.907560    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe10361-91bc-4ff6-a6d4-faec973ad665-kube-api-access-tbh4v" pod "5fe10361-91bc-4ff6-a6d4-faec973ad665" (UID: "5fe10361-91bc-4ff6-a6d4-faec973ad665"). InnerVolumeSpecName "kube-api-access-tbh4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:19:38.006720 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:38.006658    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9949e8e5-8f11-4c06-8a7b-20e8640c2ac6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9949e8e5-8f11-4c06-8a7b-20e8640c2ac6-plugins-config-volume\") pod \"9949e8e5-8f11-4c06-8a7b-20e8640c2ac6\" (UID: \"9949e8e5-8f11-4c06-8a7b-20e8640c2ac6\") "
Oct 31 01:19:38.006925 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:38.006755    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9949e8e5-8f11-4c06-8a7b-20e8640c2ac6-kube-api-access-mjlpr\" (UniqueName: \"kubernetes.io/projected/9949e8e5-8f11-4c06-8a7b-20e8640c2ac6-kube-api-access-mjlpr\") pod \"9949e8e5-8f11-4c06-8a7b-20e8640c2ac6\" (UID: \"9949e8e5-8f11-4c06-8a7b-20e8640c2ac6\") "
Oct 31 01:19:38.007131 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:38.007101    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9949e8e5-8f11-4c06-8a7b-20e8640c2ac6-plugins-config-volume" pod "9949e8e5-8f11-4c06-8a7b-20e8640c2ac6" (UID: "9949e8e5-8f11-4c06-8a7b-20e8640c2ac6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:19:38.007593 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:38.007566    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbh4v\" (UniqueName: \"kubernetes.io/projected/5fe10361-91bc-4ff6-a6d4-faec973ad665-kube-api-access-tbh4v\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:38.007593 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:38.007598    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9949e8e5-8f11-4c06-8a7b-20e8640c2ac6-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:38.009191 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:38.009164    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9949e8e5-8f11-4c06-8a7b-20e8640c2ac6-kube-api-access-mjlpr" pod "9949e8e5-8f11-4c06-8a7b-20e8640c2ac6" (UID: "9949e8e5-8f11-4c06-8a7b-20e8640c2ac6"). InnerVolumeSpecName "kube-api-access-mjlpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:19:38.108573 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:38.108539    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjlpr\" (UniqueName: \"kubernetes.io/projected/9949e8e5-8f11-4c06-8a7b-20e8640c2ac6-kube-api-access-mjlpr\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:38.181744 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:38.181706    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c"]
Oct 31 01:19:38.193943 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:38.193905    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-498580-86c7764565-fxm8c"]
Oct 31 01:19:38.255427 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:38.255385    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t" podUID="852c332c-c02b-4ccc-b4ef-8733645922c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:19:38.848560 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:38.848483    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8"
Oct 31 01:19:38.848560 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:38.848524    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8" event={"ID":"9949e8e5-8f11-4c06-8a7b-20e8640c2ac6","Type":"ContainerDied","Data":"1e3ae62ddd339686c51c1d6543977c655d08b004a719c70dedbe4f16fe45d253"}
Oct 31 01:19:38.849114 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:38.848590    4327 scope.go:122] "RemoveContainer" containerID="74253eb04a8953575096c0b5814519e61e35839ac1ee27fc56b57774965ddac1"
Oct 31 01:19:38.902172 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:38.902135    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8"]
Oct 31 01:19:38.913988 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:38.913958    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-498580-58d8d4685f-xpfv8"]
Oct 31 01:19:39.806414 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:39.806381    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n"
Oct 31 01:19:39.925182 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:39.925150    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fe10361-91bc-4ff6-a6d4-faec973ad665" path="/var/lib/kubelet/pods/5fe10361-91bc-4ff6-a6d4-faec973ad665/volumes"
Oct 31 01:19:39.925611 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:39.925593    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9949e8e5-8f11-4c06-8a7b-20e8640c2ac6" path="/var/lib/kubelet/pods/9949e8e5-8f11-4c06-8a7b-20e8640c2ac6/volumes"
Oct 31 01:19:40.269171 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:40.269136    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97" podUID="96e41c65-75e1-4501-b965-14fa687c68c9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:19:45.295923 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:19:45.295876    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:19:46.664697 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:46.664645    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"]
Oct 31 01:19:46.665115 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:46.664896    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj" podUID="d965e342-4052-4f57-a495-4d8023a1f882" containerName="istio-proxy" containerID="cri-o://9e1d1d0f26ce648fa7c79e5d512dbcbef5f13af346249bebe31800975ac20c91" gracePeriod=30
Oct 31 01:19:48.255361 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:48.255314    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t" podUID="852c332c-c02b-4ccc-b4ef-8733645922c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:19:50.269334 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:50.269293    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97" podUID="96e41c65-75e1-4501-b965-14fa687c68c9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:19:51.965211 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:51.965176    4327 generic.go:356] "Generic (PLEG): container finished" podID="d965e342-4052-4f57-a495-4d8023a1f882" containerID="9e1d1d0f26ce648fa7c79e5d512dbcbef5f13af346249bebe31800975ac20c91" exitCode=0
Oct 31 01:19:51.965585 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:51.965245    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj" event={"ID":"d965e342-4052-4f57-a495-4d8023a1f882","Type":"ContainerDied","Data":"9e1d1d0f26ce648fa7c79e5d512dbcbef5f13af346249bebe31800975ac20c91"}
Oct 31 01:19:51.965585 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:51.965334    4327 scope.go:122] "RemoveContainer" containerID="07c1a71556991ba0e4182aecd33df8c8bbb24eca38740efa9f61c6afc21a50b6"
Oct 31 01:19:51.988383 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:51.988362    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 01:19:52.073990 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.073348    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-workload-certs\") pod \"d965e342-4052-4f57-a495-4d8023a1f882\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") "
Oct 31 01:19:52.073990 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.073412    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-credential-socket\") pod \"d965e342-4052-4f57-a495-4d8023a1f882\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") "
Oct 31 01:19:52.073990 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.073464    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-istio-data\") pod \"d965e342-4052-4f57-a495-4d8023a1f882\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") "
Oct 31 01:19:52.073990 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.073571    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d965e342-4052-4f57-a495-4d8023a1f882-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d965e342-4052-4f57-a495-4d8023a1f882-istiod-ca-cert\") pod \"d965e342-4052-4f57-a495-4d8023a1f882\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") "
Oct 31 01:19:52.073990 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.073738    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d965e342-4052-4f57-a495-4d8023a1f882-istio-token\" (UniqueName: \"kubernetes.io/projected/d965e342-4052-4f57-a495-4d8023a1f882-istio-token\") pod \"d965e342-4052-4f57-a495-4d8023a1f882\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") "
Oct 31 01:19:52.073990 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.073733    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-workload-certs" pod "d965e342-4052-4f57-a495-4d8023a1f882" (UID: "d965e342-4052-4f57-a495-4d8023a1f882"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:19:52.073990 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.073789    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d965e342-4052-4f57-a495-4d8023a1f882-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d965e342-4052-4f57-a495-4d8023a1f882-istio-podinfo\") pod \"d965e342-4052-4f57-a495-4d8023a1f882\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") "
Oct 31 01:19:52.073990 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.073828    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-istio-envoy\") pod \"d965e342-4052-4f57-a495-4d8023a1f882\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") "
Oct 31 01:19:52.073990 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.073867    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d965e342-4052-4f57-a495-4d8023a1f882-kube-api-access-w9brl\" (UniqueName: \"kubernetes.io/projected/d965e342-4052-4f57-a495-4d8023a1f882-kube-api-access-w9brl\") pod \"d965e342-4052-4f57-a495-4d8023a1f882\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") "
Oct 31 01:19:52.073990 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.073900    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-workload-socket\") pod \"d965e342-4052-4f57-a495-4d8023a1f882\" (UID: \"d965e342-4052-4f57-a495-4d8023a1f882\") "
Oct 31 01:19:52.074723 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.074042    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-istio-data" pod "d965e342-4052-4f57-a495-4d8023a1f882" (UID: "d965e342-4052-4f57-a495-4d8023a1f882"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:19:52.074723 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.074274    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-credential-socket" pod "d965e342-4052-4f57-a495-4d8023a1f882" (UID: "d965e342-4052-4f57-a495-4d8023a1f882"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:19:52.074723 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.074603    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d965e342-4052-4f57-a495-4d8023a1f882-istiod-ca-cert" pod "d965e342-4052-4f57-a495-4d8023a1f882" (UID: "d965e342-4052-4f57-a495-4d8023a1f882"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:19:52.076762 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.076064    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:52.076762 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.076094    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:52.076762 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.076108    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:52.076762 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.076122    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d965e342-4052-4f57-a495-4d8023a1f882-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:52.078542 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.078505    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-workload-socket" pod "d965e342-4052-4f57-a495-4d8023a1f882" (UID: "d965e342-4052-4f57-a495-4d8023a1f882"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:19:52.083150 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.081814    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-istio-envoy" pod "d965e342-4052-4f57-a495-4d8023a1f882" (UID: "d965e342-4052-4f57-a495-4d8023a1f882"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:19:52.083150 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.081883    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d965e342-4052-4f57-a495-4d8023a1f882-kube-api-access-w9brl" pod "d965e342-4052-4f57-a495-4d8023a1f882" (UID: "d965e342-4052-4f57-a495-4d8023a1f882"). InnerVolumeSpecName "kube-api-access-w9brl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:19:52.083505 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.083484    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d965e342-4052-4f57-a495-4d8023a1f882-istio-token" pod "d965e342-4052-4f57-a495-4d8023a1f882" (UID: "d965e342-4052-4f57-a495-4d8023a1f882"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:19:52.085016 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.084994    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d965e342-4052-4f57-a495-4d8023a1f882-istio-podinfo" pod "d965e342-4052-4f57-a495-4d8023a1f882" (UID: "d965e342-4052-4f57-a495-4d8023a1f882"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:19:52.177273 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.177237    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d965e342-4052-4f57-a495-4d8023a1f882-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:52.177273 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.177271    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:52.177454 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.177281    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9brl\" (UniqueName: \"kubernetes.io/projected/d965e342-4052-4f57-a495-4d8023a1f882-kube-api-access-w9brl\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:52.177454 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.177293    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d965e342-4052-4f57-a495-4d8023a1f882-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:52.177454 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.177302    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d965e342-4052-4f57-a495-4d8023a1f882-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:52.371933 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.371903    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97"
Oct 31 01:19:52.411668 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.411640    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t"
Oct 31 01:19:52.481560 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.481469    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e41c65-75e1-4501-b965-14fa687c68c9-kube-api-access-z7gsf\" (UniqueName: \"kubernetes.io/projected/96e41c65-75e1-4501-b965-14fa687c68c9-kube-api-access-z7gsf\") pod \"96e41c65-75e1-4501-b965-14fa687c68c9\" (UID: \"96e41c65-75e1-4501-b965-14fa687c68c9\") "
Oct 31 01:19:52.481560 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.481531    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96e41c65-75e1-4501-b965-14fa687c68c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96e41c65-75e1-4501-b965-14fa687c68c9-plugins-config-volume\") pod \"96e41c65-75e1-4501-b965-14fa687c68c9\" (UID: \"96e41c65-75e1-4501-b965-14fa687c68c9\") "
Oct 31 01:19:52.481987 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.481955    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96e41c65-75e1-4501-b965-14fa687c68c9-plugins-config-volume" pod "96e41c65-75e1-4501-b965-14fa687c68c9" (UID: "96e41c65-75e1-4501-b965-14fa687c68c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:19:52.482120 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.482094    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96e41c65-75e1-4501-b965-14fa687c68c9-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:52.483751 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.483728    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e41c65-75e1-4501-b965-14fa687c68c9-kube-api-access-z7gsf" pod "96e41c65-75e1-4501-b965-14fa687c68c9" (UID: "96e41c65-75e1-4501-b965-14fa687c68c9"). InnerVolumeSpecName "kube-api-access-z7gsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:19:52.583135 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.583099    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/852c332c-c02b-4ccc-b4ef-8733645922c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/852c332c-c02b-4ccc-b4ef-8733645922c9-plugins-config-volume\") pod \"852c332c-c02b-4ccc-b4ef-8733645922c9\" (UID: \"852c332c-c02b-4ccc-b4ef-8733645922c9\") "
Oct 31 01:19:52.583335 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.583243    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/852c332c-c02b-4ccc-b4ef-8733645922c9-kube-api-access-cw4pw\" (UniqueName: \"kubernetes.io/projected/852c332c-c02b-4ccc-b4ef-8733645922c9-kube-api-access-cw4pw\") pod \"852c332c-c02b-4ccc-b4ef-8733645922c9\" (UID: \"852c332c-c02b-4ccc-b4ef-8733645922c9\") "
Oct 31 01:19:52.583501 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.583475    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/852c332c-c02b-4ccc-b4ef-8733645922c9-plugins-config-volume" pod "852c332c-c02b-4ccc-b4ef-8733645922c9" (UID: "852c332c-c02b-4ccc-b4ef-8733645922c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:19:52.583926 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.583908    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/852c332c-c02b-4ccc-b4ef-8733645922c9-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:52.583977 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.583929    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7gsf\" (UniqueName: \"kubernetes.io/projected/96e41c65-75e1-4501-b965-14fa687c68c9-kube-api-access-z7gsf\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:52.585481 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.585460    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852c332c-c02b-4ccc-b4ef-8733645922c9-kube-api-access-cw4pw" pod "852c332c-c02b-4ccc-b4ef-8733645922c9" (UID: "852c332c-c02b-4ccc-b4ef-8733645922c9"). InnerVolumeSpecName "kube-api-access-cw4pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:19:52.685379 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.685337    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw4pw\" (UniqueName: \"kubernetes.io/projected/852c332c-c02b-4ccc-b4ef-8733645922c9-kube-api-access-cw4pw\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:19:52.973893 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.973856    4327 generic.go:356] "Generic (PLEG): container finished" podID="96e41c65-75e1-4501-b965-14fa687c68c9" containerID="123afc8e585a785406de80a2f2a1de1cbb1f8b79e8f3508795ee8e4a5a303904" exitCode=0
Oct 31 01:19:52.974376 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.973904    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97"
Oct 31 01:19:52.974376 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.973939    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97" event={"ID":"96e41c65-75e1-4501-b965-14fa687c68c9","Type":"ContainerDied","Data":"123afc8e585a785406de80a2f2a1de1cbb1f8b79e8f3508795ee8e4a5a303904"}
Oct 31 01:19:52.974376 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.974004    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97" event={"ID":"96e41c65-75e1-4501-b965-14fa687c68c9","Type":"ContainerDied","Data":"8b2e26a103a7e253374f24df1d115dc0169882fb3270616a24c64343589d6a60"}
Oct 31 01:19:52.974376 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.974039    4327 scope.go:122] "RemoveContainer" containerID="123afc8e585a785406de80a2f2a1de1cbb1f8b79e8f3508795ee8e4a5a303904"
Oct 31 01:19:52.975595 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.975578    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"
Oct 31 01:19:52.975709 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.975616    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj" event={"ID":"d965e342-4052-4f57-a495-4d8023a1f882","Type":"ContainerDied","Data":"6b362cad5b33ac96f757b07b0dacfeda577465ef3f08128bd1c30e3f2ae7b7df"}
Oct 31 01:19:52.977222 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.977172    4327 generic.go:356] "Generic (PLEG): container finished" podID="852c332c-c02b-4ccc-b4ef-8733645922c9" containerID="ad07de1e5863ffa3fa213ca07b65e6868c2974aa5977ff4dab513c593e426427" exitCode=0
Oct 31 01:19:52.977222 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.977203    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t"
Oct 31 01:19:52.977391 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.977203    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t" event={"ID":"852c332c-c02b-4ccc-b4ef-8733645922c9","Type":"ContainerDied","Data":"ad07de1e5863ffa3fa213ca07b65e6868c2974aa5977ff4dab513c593e426427"}
Oct 31 01:19:52.977391 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.977339    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t" event={"ID":"852c332c-c02b-4ccc-b4ef-8733645922c9","Type":"ContainerDied","Data":"bea9f43228facc142dea313d29660002cdcb0a2f8122da2c845531b91a9a2037"}
Oct 31 01:19:52.987880 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.987742    4327 scope.go:122] "RemoveContainer" containerID="123afc8e585a785406de80a2f2a1de1cbb1f8b79e8f3508795ee8e4a5a303904"
Oct 31 01:19:52.988083 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:19:52.988058    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"123afc8e585a785406de80a2f2a1de1cbb1f8b79e8f3508795ee8e4a5a303904\": container with ID starting with 123afc8e585a785406de80a2f2a1de1cbb1f8b79e8f3508795ee8e4a5a303904 not found: ID does not exist" containerID="123afc8e585a785406de80a2f2a1de1cbb1f8b79e8f3508795ee8e4a5a303904"
Oct 31 01:19:52.988172 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.988096    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"123afc8e585a785406de80a2f2a1de1cbb1f8b79e8f3508795ee8e4a5a303904"} err="failed to get container status \"123afc8e585a785406de80a2f2a1de1cbb1f8b79e8f3508795ee8e4a5a303904\": rpc error: code = NotFound desc = could not find container \"123afc8e585a785406de80a2f2a1de1cbb1f8b79e8f3508795ee8e4a5a303904\": container with ID starting with 123afc8e585a785406de80a2f2a1de1cbb1f8b79e8f3508795ee8e4a5a303904 not found: ID does not exist"
Oct 31 01:19:52.988172 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.988133    4327 scope.go:122] "RemoveContainer" containerID="9e1d1d0f26ce648fa7c79e5d512dbcbef5f13af346249bebe31800975ac20c91"
Oct 31 01:19:52.998300 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:52.998282    4327 scope.go:122] "RemoveContainer" containerID="ad07de1e5863ffa3fa213ca07b65e6868c2974aa5977ff4dab513c593e426427"
Oct 31 01:19:53.008168 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:53.008148    4327 scope.go:122] "RemoveContainer" containerID="ad07de1e5863ffa3fa213ca07b65e6868c2974aa5977ff4dab513c593e426427"
Oct 31 01:19:53.008455 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:19:53.008428    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad07de1e5863ffa3fa213ca07b65e6868c2974aa5977ff4dab513c593e426427\": container with ID starting with ad07de1e5863ffa3fa213ca07b65e6868c2974aa5977ff4dab513c593e426427 not found: ID does not exist" containerID="ad07de1e5863ffa3fa213ca07b65e6868c2974aa5977ff4dab513c593e426427"
Oct 31 01:19:53.008556 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:53.008464    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad07de1e5863ffa3fa213ca07b65e6868c2974aa5977ff4dab513c593e426427"} err="failed to get container status \"ad07de1e5863ffa3fa213ca07b65e6868c2974aa5977ff4dab513c593e426427\": rpc error: code = NotFound desc = could not find container \"ad07de1e5863ffa3fa213ca07b65e6868c2974aa5977ff4dab513c593e426427\": container with ID starting with ad07de1e5863ffa3fa213ca07b65e6868c2974aa5977ff4dab513c593e426427 not found: ID does not exist"
Oct 31 01:19:53.091373 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:53.091338    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t"]
Oct 31 01:19:53.100053 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:53.100023    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b8fd5a-75ddf6c547-kdn6t"]
Oct 31 01:19:53.134726 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:53.134650    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"]
Oct 31 01:19:53.146509 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:53.146476    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-sp5kj"]
Oct 31 01:19:53.172894 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:53.172862    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97"]
Oct 31 01:19:53.177055 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:53.177022    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-b8fd5a-65bdff79c-kwn97"]
Oct 31 01:19:53.934175 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:53.934136    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="852c332c-c02b-4ccc-b4ef-8733645922c9" path="/var/lib/kubelet/pods/852c332c-c02b-4ccc-b4ef-8733645922c9/volumes"
Oct 31 01:19:53.934775 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:53.934758    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96e41c65-75e1-4501-b965-14fa687c68c9" path="/var/lib/kubelet/pods/96e41c65-75e1-4501-b965-14fa687c68c9/volumes"
Oct 31 01:19:53.935206 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:53.935193    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d965e342-4052-4f57-a495-4d8023a1f882" path="/var/lib/kubelet/pods/d965e342-4052-4f57-a495-4d8023a1f882/volumes"
Oct 31 01:19:57.814100 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:57.814057    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk"]
Oct 31 01:19:57.835190 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:19:57.835148    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt"]
Oct 31 01:20:01.079800 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:01.079758    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"]
Oct 31 01:20:01.080341 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:01.080031    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7" podUID="4c28e892-156c-4df2-a55e-af21750b4fda" containerName="istio-proxy" containerID="cri-o://8b1e61b00df317da699357dcc29e787ad6a04e8268b1ead678c1a55ffc3e7f12" gracePeriod=30
Oct 31 01:20:02.647482 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:02.647442    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2"]
Oct 31 01:20:02.648029 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:02.647755    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2" podUID="55c787a6-2c98-4474-ac32-82f652780c14" containerName="maas-api" containerID="cri-o://36c6257e79f22ebfd38d8b7858890d9ade7839f3b77eea8e5ca78585bf9cd9aa" gracePeriod=30
Oct 31 01:20:02.815004 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:02.814954    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk" podUID="67146ff0-b14c-4bb9-9a6b-db437c984bbd" containerName="payload-processing" containerID="cri-o://cc12fda3ec16bcf761da60cd3a1fc701c6e66b4b97850d583f7b74d429edd4a5" gracePeriod=25
Oct 31 01:20:02.825277 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:02.825245    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv"]
Oct 31 01:20:02.825485 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:02.825460    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv" podUID="2c2c3784-1773-40a2-b373-f925c7e4d239" containerName="payload-processing" containerID="cri-o://a649def0154655d5bf2311c48de1bb7457eb5d7f726de8fb66f12be3ccdc23d2" gracePeriod=30
Oct 31 01:20:02.836161 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:02.836106    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt" podUID="b302fecf-fc39-4122-bc35-83585cc68ae5" containerName="payload-pre-processing" containerID="cri-o://e90f60efab88ef607254881b9b829967130a5a8c64e46ace7fb6dcbb0eabdc6f" gracePeriod=25
Oct 31 01:20:02.859787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:02.859739    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8"]
Oct 31 01:20:02.860064 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:02.860037    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8" podUID="e576a213-4352-4cea-b68a-388edabf4ab0" containerName="payload-pre-processing" containerID="cri-o://00bd701edc5adae6316d7017d11770333ab84c71b7ae2aa617ac636400ea1ab2" gracePeriod=30
Oct 31 01:20:02.920928 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:02.920888    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv" podUID="2c2c3784-1773-40a2-b373-f925c7e4d239" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.229:9004: connect: connection refused"
Oct 31 01:20:02.921528 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:02.921457    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8" podUID="e576a213-4352-4cea-b68a-388edabf4ab0" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.228:9004: connect: connection refused"
Oct 31 01:20:02.991747 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:02.991719    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2"
Oct 31 01:20:03.075357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.074413    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55c787a6-2c98-4474-ac32-82f652780c14-metrics-certs\" (UniqueName: \"kubernetes.io/secret/55c787a6-2c98-4474-ac32-82f652780c14-metrics-certs\") pod \"55c787a6-2c98-4474-ac32-82f652780c14\" (UID: \"55c787a6-2c98-4474-ac32-82f652780c14\") "
Oct 31 01:20:03.075357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.074544    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55c787a6-2c98-4474-ac32-82f652780c14-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55c787a6-2c98-4474-ac32-82f652780c14-maas-api-tls\") pod \"55c787a6-2c98-4474-ac32-82f652780c14\" (UID: \"55c787a6-2c98-4474-ac32-82f652780c14\") "
Oct 31 01:20:03.075357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.074607    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55c787a6-2c98-4474-ac32-82f652780c14-kube-api-access-rd579\" (UniqueName: \"kubernetes.io/projected/55c787a6-2c98-4474-ac32-82f652780c14-kube-api-access-rd579\") pod \"55c787a6-2c98-4474-ac32-82f652780c14\" (UID: \"55c787a6-2c98-4474-ac32-82f652780c14\") "
Oct 31 01:20:03.075357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.074759    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55c787a6-2c98-4474-ac32-82f652780c14-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c787a6-2c98-4474-ac32-82f652780c14-openshift-service-ca\") pod \"55c787a6-2c98-4474-ac32-82f652780c14\" (UID: \"55c787a6-2c98-4474-ac32-82f652780c14\") "
Oct 31 01:20:03.075918 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.075574    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c787a6-2c98-4474-ac32-82f652780c14-openshift-service-ca" pod "55c787a6-2c98-4474-ac32-82f652780c14" (UID: "55c787a6-2c98-4474-ac32-82f652780c14"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:20:03.081064 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.081033    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c787a6-2c98-4474-ac32-82f652780c14-kube-api-access-rd579" pod "55c787a6-2c98-4474-ac32-82f652780c14" (UID: "55c787a6-2c98-4474-ac32-82f652780c14"). InnerVolumeSpecName "kube-api-access-rd579". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:20:03.087254 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.084084    4327 generic.go:356] "Generic (PLEG): container finished" podID="55c787a6-2c98-4474-ac32-82f652780c14" containerID="36c6257e79f22ebfd38d8b7858890d9ade7839f3b77eea8e5ca78585bf9cd9aa" exitCode=0
Oct 31 01:20:03.087254 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.084228    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2" event={"ID":"55c787a6-2c98-4474-ac32-82f652780c14","Type":"ContainerDied","Data":"36c6257e79f22ebfd38d8b7858890d9ade7839f3b77eea8e5ca78585bf9cd9aa"}
Oct 31 01:20:03.087254 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.084281    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2" event={"ID":"55c787a6-2c98-4474-ac32-82f652780c14","Type":"ContainerDied","Data":"ddd9239282420697311dad97d97e7b86033c65f1cf3c3e6cc58a1ff8c49e41fd"}
Oct 31 01:20:03.087254 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.084310    4327 scope.go:122] "RemoveContainer" containerID="36c6257e79f22ebfd38d8b7858890d9ade7839f3b77eea8e5ca78585bf9cd9aa"
Oct 31 01:20:03.087254 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.084476    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2"
Oct 31 01:20:03.087254 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.087005    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55c787a6-2c98-4474-ac32-82f652780c14-maas-api-tls" pod "55c787a6-2c98-4474-ac32-82f652780c14" (UID: "55c787a6-2c98-4474-ac32-82f652780c14"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:20:03.098844 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.089858    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv" event={"ID":"2c2c3784-1773-40a2-b373-f925c7e4d239","Type":"ContainerDied","Data":"a649def0154655d5bf2311c48de1bb7457eb5d7f726de8fb66f12be3ccdc23d2"}
Oct 31 01:20:03.098844 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.089962    4327 generic.go:356] "Generic (PLEG): container finished" podID="2c2c3784-1773-40a2-b373-f925c7e4d239" containerID="a649def0154655d5bf2311c48de1bb7457eb5d7f726de8fb66f12be3ccdc23d2" exitCode=0
Oct 31 01:20:03.098844 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.091067    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55c787a6-2c98-4474-ac32-82f652780c14-metrics-certs" pod "55c787a6-2c98-4474-ac32-82f652780c14" (UID: "55c787a6-2c98-4474-ac32-82f652780c14"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:20:03.098844 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.092554    4327 generic.go:356] "Generic (PLEG): container finished" podID="e576a213-4352-4cea-b68a-388edabf4ab0" containerID="00bd701edc5adae6316d7017d11770333ab84c71b7ae2aa617ac636400ea1ab2" exitCode=0
Oct 31 01:20:03.098844 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.092672    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8" event={"ID":"e576a213-4352-4cea-b68a-388edabf4ab0","Type":"ContainerDied","Data":"00bd701edc5adae6316d7017d11770333ab84c71b7ae2aa617ac636400ea1ab2"}
Oct 31 01:20:03.112361 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.111914    4327 scope.go:122] "RemoveContainer" containerID="36c6257e79f22ebfd38d8b7858890d9ade7839f3b77eea8e5ca78585bf9cd9aa"
Oct 31 01:20:03.113159 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:20:03.112647    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c6257e79f22ebfd38d8b7858890d9ade7839f3b77eea8e5ca78585bf9cd9aa\": container with ID starting with 36c6257e79f22ebfd38d8b7858890d9ade7839f3b77eea8e5ca78585bf9cd9aa not found: ID does not exist" containerID="36c6257e79f22ebfd38d8b7858890d9ade7839f3b77eea8e5ca78585bf9cd9aa"
Oct 31 01:20:03.113159 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.112700    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c6257e79f22ebfd38d8b7858890d9ade7839f3b77eea8e5ca78585bf9cd9aa"} err="failed to get container status \"36c6257e79f22ebfd38d8b7858890d9ade7839f3b77eea8e5ca78585bf9cd9aa\": rpc error: code = NotFound desc = could not find container \"36c6257e79f22ebfd38d8b7858890d9ade7839f3b77eea8e5ca78585bf9cd9aa\": container with ID starting with 36c6257e79f22ebfd38d8b7858890d9ade7839f3b77eea8e5ca78585bf9cd9aa not found: ID does not exist"
Oct 31 01:20:03.146836 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.146753    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv"
Oct 31 01:20:03.206776 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.190134    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd579\" (UniqueName: \"kubernetes.io/projected/55c787a6-2c98-4474-ac32-82f652780c14-kube-api-access-rd579\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:03.206776 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.190173    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c787a6-2c98-4474-ac32-82f652780c14-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:03.206776 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.190187    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55c787a6-2c98-4474-ac32-82f652780c14-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:03.206776 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.190200    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55c787a6-2c98-4474-ac32-82f652780c14-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:03.291020 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.290984    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c2c3784-1773-40a2-b373-f925c7e4d239-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c2c3784-1773-40a2-b373-f925c7e4d239-plugins-config-volume\") pod \"2c2c3784-1773-40a2-b373-f925c7e4d239\" (UID: \"2c2c3784-1773-40a2-b373-f925c7e4d239\") "
Oct 31 01:20:03.291261 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.291033    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c2c3784-1773-40a2-b373-f925c7e4d239-kube-api-access-7gmkr\" (UniqueName: \"kubernetes.io/projected/2c2c3784-1773-40a2-b373-f925c7e4d239-kube-api-access-7gmkr\") pod \"2c2c3784-1773-40a2-b373-f925c7e4d239\" (UID: \"2c2c3784-1773-40a2-b373-f925c7e4d239\") "
Oct 31 01:20:03.291406 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.291385    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c2c3784-1773-40a2-b373-f925c7e4d239-plugins-config-volume" pod "2c2c3784-1773-40a2-b373-f925c7e4d239" (UID: "2c2c3784-1773-40a2-b373-f925c7e4d239"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:20:03.291631 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.291596    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c2c3784-1773-40a2-b373-f925c7e4d239-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:03.293723 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.293667    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2c3784-1773-40a2-b373-f925c7e4d239-kube-api-access-7gmkr" pod "2c2c3784-1773-40a2-b373-f925c7e4d239" (UID: "2c2c3784-1773-40a2-b373-f925c7e4d239"). InnerVolumeSpecName "kube-api-access-7gmkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:20:03.301783 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.301758    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8"
Oct 31 01:20:03.392648 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.392607    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e576a213-4352-4cea-b68a-388edabf4ab0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e576a213-4352-4cea-b68a-388edabf4ab0-plugins-config-volume\") pod \"e576a213-4352-4cea-b68a-388edabf4ab0\" (UID: \"e576a213-4352-4cea-b68a-388edabf4ab0\") "
Oct 31 01:20:03.392837 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.392666    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e576a213-4352-4cea-b68a-388edabf4ab0-kube-api-access-mm5rz\" (UniqueName: \"kubernetes.io/projected/e576a213-4352-4cea-b68a-388edabf4ab0-kube-api-access-mm5rz\") pod \"e576a213-4352-4cea-b68a-388edabf4ab0\" (UID: \"e576a213-4352-4cea-b68a-388edabf4ab0\") "
Oct 31 01:20:03.393459 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.393430    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gmkr\" (UniqueName: \"kubernetes.io/projected/2c2c3784-1773-40a2-b373-f925c7e4d239-kube-api-access-7gmkr\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:03.393562 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.393458    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e576a213-4352-4cea-b68a-388edabf4ab0-plugins-config-volume" pod "e576a213-4352-4cea-b68a-388edabf4ab0" (UID: "e576a213-4352-4cea-b68a-388edabf4ab0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:20:03.397467 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.397436    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e576a213-4352-4cea-b68a-388edabf4ab0-kube-api-access-mm5rz" pod "e576a213-4352-4cea-b68a-388edabf4ab0" (UID: "e576a213-4352-4cea-b68a-388edabf4ab0"). InnerVolumeSpecName "kube-api-access-mm5rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:20:03.434502 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.434467    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2"]
Oct 31 01:20:03.438941 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.438914    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa97ea-6549cf8dc5-gcwx2"]
Oct 31 01:20:03.494848 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.494751    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e576a213-4352-4cea-b68a-388edabf4ab0-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:03.494848 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.494782    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm5rz\" (UniqueName: \"kubernetes.io/projected/e576a213-4352-4cea-b68a-388edabf4ab0-kube-api-access-mm5rz\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:03.922311 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:03.922273    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55c787a6-2c98-4474-ac32-82f652780c14" path="/var/lib/kubelet/pods/55c787a6-2c98-4474-ac32-82f652780c14/volumes"
Oct 31 01:20:04.101648 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:04.101611    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv"
Oct 31 01:20:04.101648 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:04.101628    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv" event={"ID":"2c2c3784-1773-40a2-b373-f925c7e4d239","Type":"ContainerDied","Data":"8a252ad83fe3f02972041550d07bbba250724b5d5f1e49941b0078770c9cd214"}
Oct 31 01:20:04.101969 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:04.101716    4327 scope.go:122] "RemoveContainer" containerID="a649def0154655d5bf2311c48de1bb7457eb5d7f726de8fb66f12be3ccdc23d2"
Oct 31 01:20:04.103117 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:04.103096    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8"
Oct 31 01:20:04.103190 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:04.103128    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8" event={"ID":"e576a213-4352-4cea-b68a-388edabf4ab0","Type":"ContainerDied","Data":"67a2a62e785e24c0f4bb86412a90847edfdfaea6b14ab62cf97bed5f7baa9385"}
Oct 31 01:20:04.112721 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:04.112700    4327 scope.go:122] "RemoveContainer" containerID="00bd701edc5adae6316d7017d11770333ab84c71b7ae2aa617ac636400ea1ab2"
Oct 31 01:20:04.154855 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:04.154817    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv"]
Oct 31 01:20:04.163790 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:04.163759    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-7748477fb4-gd8kv"]
Oct 31 01:20:04.197267 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:04.197188    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8"]
Oct 31 01:20:04.203758 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:04.203729    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-67bbf6f975-f8nm8"]
Oct 31 01:20:04.346772 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:04.346738    4327 scope.go:122] "RemoveContainer" containerID="cf063eb96950b0bc3b361f4c589df04e4b5549f6c45693c751c9dd5cce14c418"
Oct 31 01:20:05.922896 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:05.922863    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c2c3784-1773-40a2-b373-f925c7e4d239" path="/var/lib/kubelet/pods/2c2c3784-1773-40a2-b373-f925c7e4d239/volumes"
Oct 31 01:20:05.923393 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:05.923374    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e576a213-4352-4cea-b68a-388edabf4ab0" path="/var/lib/kubelet/pods/e576a213-4352-4cea-b68a-388edabf4ab0/volumes"
Oct 31 01:20:06.363885 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.363855    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:20:06.431340 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.431303    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-istio-envoy\") pod \"4c28e892-156c-4df2-a55e-af21750b4fda\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") "
Oct 31 01:20:06.431548 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.431432    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-credential-socket\") pod \"4c28e892-156c-4df2-a55e-af21750b4fda\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") "
Oct 31 01:20:06.431631 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.431551    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c28e892-156c-4df2-a55e-af21750b4fda-istio-token\" (UniqueName: \"kubernetes.io/projected/4c28e892-156c-4df2-a55e-af21750b4fda-istio-token\") pod \"4c28e892-156c-4df2-a55e-af21750b4fda\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") "
Oct 31 01:20:06.431631 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.431594    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c28e892-156c-4df2-a55e-af21750b4fda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c28e892-156c-4df2-a55e-af21750b4fda-istio-podinfo\") pod \"4c28e892-156c-4df2-a55e-af21750b4fda\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") "
Oct 31 01:20:06.431631 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.431624    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c28e892-156c-4df2-a55e-af21750b4fda-kube-api-access-fg9gq\" (UniqueName: \"kubernetes.io/projected/4c28e892-156c-4df2-a55e-af21750b4fda-kube-api-access-fg9gq\") pod \"4c28e892-156c-4df2-a55e-af21750b4fda\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") "
Oct 31 01:20:06.431834 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.431699    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c28e892-156c-4df2-a55e-af21750b4fda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c28e892-156c-4df2-a55e-af21750b4fda-istiod-ca-cert\") pod \"4c28e892-156c-4df2-a55e-af21750b4fda\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") "
Oct 31 01:20:06.431834 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.431770    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-workload-socket\") pod \"4c28e892-156c-4df2-a55e-af21750b4fda\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") "
Oct 31 01:20:06.431834 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.431807    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-credential-socket" pod "4c28e892-156c-4df2-a55e-af21750b4fda" (UID: "4c28e892-156c-4df2-a55e-af21750b4fda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:06.431998 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.431850    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-workload-certs\") pod \"4c28e892-156c-4df2-a55e-af21750b4fda\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") "
Oct 31 01:20:06.433694 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.433081    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c28e892-156c-4df2-a55e-af21750b4fda-istiod-ca-cert" pod "4c28e892-156c-4df2-a55e-af21750b4fda" (UID: "4c28e892-156c-4df2-a55e-af21750b4fda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:20:06.433694 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.433285    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-istio-data" pod "4c28e892-156c-4df2-a55e-af21750b4fda" (UID: "4c28e892-156c-4df2-a55e-af21750b4fda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:06.434188 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.434160    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-workload-certs" pod "4c28e892-156c-4df2-a55e-af21750b4fda" (UID: "4c28e892-156c-4df2-a55e-af21750b4fda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:06.434319 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.434227    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-workload-socket" pod "4c28e892-156c-4df2-a55e-af21750b4fda" (UID: "4c28e892-156c-4df2-a55e-af21750b4fda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:06.434319 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.431937    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-istio-data\") pod \"4c28e892-156c-4df2-a55e-af21750b4fda\" (UID: \"4c28e892-156c-4df2-a55e-af21750b4fda\") "
Oct 31 01:20:06.434940 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.434921    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c28e892-156c-4df2-a55e-af21750b4fda-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:06.435050 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.434945    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:06.435050 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.434961    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:06.435050 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.434975    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:06.435050 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.434987    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:06.442891 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.436475    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-istio-envoy" pod "4c28e892-156c-4df2-a55e-af21750b4fda" (UID: "4c28e892-156c-4df2-a55e-af21750b4fda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:06.442891 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.436824    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c28e892-156c-4df2-a55e-af21750b4fda-kube-api-access-fg9gq" pod "4c28e892-156c-4df2-a55e-af21750b4fda" (UID: "4c28e892-156c-4df2-a55e-af21750b4fda"). InnerVolumeSpecName "kube-api-access-fg9gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:20:06.442891 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.436950    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c28e892-156c-4df2-a55e-af21750b4fda-istio-podinfo" pod "4c28e892-156c-4df2-a55e-af21750b4fda" (UID: "4c28e892-156c-4df2-a55e-af21750b4fda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:20:06.442891 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.436985    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c28e892-156c-4df2-a55e-af21750b4fda-istio-token" pod "4c28e892-156c-4df2-a55e-af21750b4fda" (UID: "4c28e892-156c-4df2-a55e-af21750b4fda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:20:06.502706 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.502607    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt" podUID="b302fecf-fc39-4122-bc35-83585cc68ae5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:20:06.502706 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.502622    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk" podUID="67146ff0-b14c-4bb9-9a6b-db437c984bbd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:20:06.535993 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.535958    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c28e892-156c-4df2-a55e-af21750b4fda-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:06.535993 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.535993    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c28e892-156c-4df2-a55e-af21750b4fda-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:06.536210 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.536003    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c28e892-156c-4df2-a55e-af21750b4fda-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:06.536210 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:06.536018    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg9gq\" (UniqueName: \"kubernetes.io/projected/4c28e892-156c-4df2-a55e-af21750b4fda-kube-api-access-fg9gq\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:07.123560 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:07.123516    4327 generic.go:356] "Generic (PLEG): container finished" podID="4c28e892-156c-4df2-a55e-af21750b4fda" containerID="8b1e61b00df317da699357dcc29e787ad6a04e8268b1ead678c1a55ffc3e7f12" exitCode=0
Oct 31 01:20:07.124089 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:07.123574    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"
Oct 31 01:20:07.124089 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:07.123607    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7" event={"ID":"4c28e892-156c-4df2-a55e-af21750b4fda","Type":"ContainerDied","Data":"8b1e61b00df317da699357dcc29e787ad6a04e8268b1ead678c1a55ffc3e7f12"}
Oct 31 01:20:07.124089 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:07.123659    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7" event={"ID":"4c28e892-156c-4df2-a55e-af21750b4fda","Type":"ContainerDied","Data":"05a0ab5a1c336735c5200faa896c617cfe6035fe87191da1ddc765885deb993f"}
Oct 31 01:20:07.124089 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:07.123715    4327 scope.go:122] "RemoveContainer" containerID="8b1e61b00df317da699357dcc29e787ad6a04e8268b1ead678c1a55ffc3e7f12"
Oct 31 01:20:07.136286 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:07.136265    4327 scope.go:122] "RemoveContainer" containerID="8b1e61b00df317da699357dcc29e787ad6a04e8268b1ead678c1a55ffc3e7f12"
Oct 31 01:20:07.136557 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:20:07.136536    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b1e61b00df317da699357dcc29e787ad6a04e8268b1ead678c1a55ffc3e7f12\": container with ID starting with 8b1e61b00df317da699357dcc29e787ad6a04e8268b1ead678c1a55ffc3e7f12 not found: ID does not exist" containerID="8b1e61b00df317da699357dcc29e787ad6a04e8268b1ead678c1a55ffc3e7f12"
Oct 31 01:20:07.136606 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:07.136569    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b1e61b00df317da699357dcc29e787ad6a04e8268b1ead678c1a55ffc3e7f12"} err="failed to get container status \"8b1e61b00df317da699357dcc29e787ad6a04e8268b1ead678c1a55ffc3e7f12\": rpc error: code = NotFound desc = could not find container \"8b1e61b00df317da699357dcc29e787ad6a04e8268b1ead678c1a55ffc3e7f12\": container with ID starting with 8b1e61b00df317da699357dcc29e787ad6a04e8268b1ead678c1a55ffc3e7f12 not found: ID does not exist"
Oct 31 01:20:07.190007 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:07.189962    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"]
Oct 31 01:20:07.196674 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:07.196643    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-fpbk7"]
Oct 31 01:20:07.921789 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:07.921753    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c28e892-156c-4df2-a55e-af21750b4fda" path="/var/lib/kubelet/pods/4c28e892-156c-4df2-a55e-af21750b4fda/volumes"
Oct 31 01:20:15.293610 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:20:15.293562    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:20:16.502098 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:16.502055    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk" podUID="67146ff0-b14c-4bb9-9a6b-db437c984bbd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:20:16.502526 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:16.502055    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt" podUID="b302fecf-fc39-4122-bc35-83585cc68ae5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:20:19.510536 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:19.510505    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq_9ec0db43-a0c1-4e82-970b-dbe3f5585acd/payload-processing/0.log"
Oct 31 01:20:20.182312 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:20.182281    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr_fdeebb2d-1d8b-49e0-bdee-af282e78e7a1/payload-processing/0.log"
Oct 31 01:20:20.432222 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:20.432189    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr_fdeebb2d-1d8b-49e0-bdee-af282e78e7a1/payload-processing/0.log"
Oct 31 01:20:23.139862 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.139827    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk"
Oct 31 01:20:23.157550 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.157525    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67146ff0-b14c-4bb9-9a6b-db437c984bbd-kube-api-access-wkjzc\" (UniqueName: \"kubernetes.io/projected/67146ff0-b14c-4bb9-9a6b-db437c984bbd-kube-api-access-wkjzc\") pod \"67146ff0-b14c-4bb9-9a6b-db437c984bbd\" (UID: \"67146ff0-b14c-4bb9-9a6b-db437c984bbd\") "
Oct 31 01:20:23.157788 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.157769    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67146ff0-b14c-4bb9-9a6b-db437c984bbd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67146ff0-b14c-4bb9-9a6b-db437c984bbd-plugins-config-volume\") pod \"67146ff0-b14c-4bb9-9a6b-db437c984bbd\" (UID: \"67146ff0-b14c-4bb9-9a6b-db437c984bbd\") "
Oct 31 01:20:23.158551 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.158524    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67146ff0-b14c-4bb9-9a6b-db437c984bbd-plugins-config-volume" pod "67146ff0-b14c-4bb9-9a6b-db437c984bbd" (UID: "67146ff0-b14c-4bb9-9a6b-db437c984bbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:20:23.159761 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.159741    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67146ff0-b14c-4bb9-9a6b-db437c984bbd-kube-api-access-wkjzc" pod "67146ff0-b14c-4bb9-9a6b-db437c984bbd" (UID: "67146ff0-b14c-4bb9-9a6b-db437c984bbd"). InnerVolumeSpecName "kube-api-access-wkjzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:20:23.168960 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.168941    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt"
Oct 31 01:20:23.239573 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.239539    4327 generic.go:356] "Generic (PLEG): container finished" podID="67146ff0-b14c-4bb9-9a6b-db437c984bbd" containerID="cc12fda3ec16bcf761da60cd3a1fc701c6e66b4b97850d583f7b74d429edd4a5" exitCode=0
Oct 31 01:20:23.239796 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.239596    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk"
Oct 31 01:20:23.239796 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.239625    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk" event={"ID":"67146ff0-b14c-4bb9-9a6b-db437c984bbd","Type":"ContainerDied","Data":"cc12fda3ec16bcf761da60cd3a1fc701c6e66b4b97850d583f7b74d429edd4a5"}
Oct 31 01:20:23.239796 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.239747    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk" event={"ID":"67146ff0-b14c-4bb9-9a6b-db437c984bbd","Type":"ContainerDied","Data":"94eebf8eb4faf02a8022c7597c5867b7d3e216a5ab843b79d29d465af827d5c5"}
Oct 31 01:20:23.239796 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.239779    4327 scope.go:122] "RemoveContainer" containerID="cc12fda3ec16bcf761da60cd3a1fc701c6e66b4b97850d583f7b74d429edd4a5"
Oct 31 01:20:23.241477 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.241395    4327 generic.go:356] "Generic (PLEG): container finished" podID="b302fecf-fc39-4122-bc35-83585cc68ae5" containerID="e90f60efab88ef607254881b9b829967130a5a8c64e46ace7fb6dcbb0eabdc6f" exitCode=0
Oct 31 01:20:23.241477 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.241436    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt" event={"ID":"b302fecf-fc39-4122-bc35-83585cc68ae5","Type":"ContainerDied","Data":"e90f60efab88ef607254881b9b829967130a5a8c64e46ace7fb6dcbb0eabdc6f"}
Oct 31 01:20:23.241477 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.241449    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt"
Oct 31 01:20:23.241477 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.241478    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt" event={"ID":"b302fecf-fc39-4122-bc35-83585cc68ae5","Type":"ContainerDied","Data":"559f3fb132fdafcead3ff6e506fc524f66622a30ad64eccc45e8e4661f25678d"}
Oct 31 01:20:23.251823 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.251805    4327 scope.go:122] "RemoveContainer" containerID="cc12fda3ec16bcf761da60cd3a1fc701c6e66b4b97850d583f7b74d429edd4a5"
Oct 31 01:20:23.252132 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:20:23.252109    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc12fda3ec16bcf761da60cd3a1fc701c6e66b4b97850d583f7b74d429edd4a5\": container with ID starting with cc12fda3ec16bcf761da60cd3a1fc701c6e66b4b97850d583f7b74d429edd4a5 not found: ID does not exist" containerID="cc12fda3ec16bcf761da60cd3a1fc701c6e66b4b97850d583f7b74d429edd4a5"
Oct 31 01:20:23.252239 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.252140    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc12fda3ec16bcf761da60cd3a1fc701c6e66b4b97850d583f7b74d429edd4a5"} err="failed to get container status \"cc12fda3ec16bcf761da60cd3a1fc701c6e66b4b97850d583f7b74d429edd4a5\": rpc error: code = NotFound desc = could not find container \"cc12fda3ec16bcf761da60cd3a1fc701c6e66b4b97850d583f7b74d429edd4a5\": container with ID starting with cc12fda3ec16bcf761da60cd3a1fc701c6e66b4b97850d583f7b74d429edd4a5 not found: ID does not exist"
Oct 31 01:20:23.252239 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.252158    4327 scope.go:122] "RemoveContainer" containerID="e90f60efab88ef607254881b9b829967130a5a8c64e46ace7fb6dcbb0eabdc6f"
Oct 31 01:20:23.259059 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.259035    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b302fecf-fc39-4122-bc35-83585cc68ae5-kube-api-access-bnmk2\" (UniqueName: \"kubernetes.io/projected/b302fecf-fc39-4122-bc35-83585cc68ae5-kube-api-access-bnmk2\") pod \"b302fecf-fc39-4122-bc35-83585cc68ae5\" (UID: \"b302fecf-fc39-4122-bc35-83585cc68ae5\") "
Oct 31 01:20:23.259187 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.259179    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b302fecf-fc39-4122-bc35-83585cc68ae5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b302fecf-fc39-4122-bc35-83585cc68ae5-plugins-config-volume\") pod \"b302fecf-fc39-4122-bc35-83585cc68ae5\" (UID: \"b302fecf-fc39-4122-bc35-83585cc68ae5\") "
Oct 31 01:20:23.259650 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.259624    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b302fecf-fc39-4122-bc35-83585cc68ae5-plugins-config-volume" pod "b302fecf-fc39-4122-bc35-83585cc68ae5" (UID: "b302fecf-fc39-4122-bc35-83585cc68ae5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:20:23.259650 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.259637    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67146ff0-b14c-4bb9-9a6b-db437c984bbd-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:23.259845 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.259661    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkjzc\" (UniqueName: \"kubernetes.io/projected/67146ff0-b14c-4bb9-9a6b-db437c984bbd-kube-api-access-wkjzc\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:23.261589 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.261563    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b302fecf-fc39-4122-bc35-83585cc68ae5-kube-api-access-bnmk2" pod "b302fecf-fc39-4122-bc35-83585cc68ae5" (UID: "b302fecf-fc39-4122-bc35-83585cc68ae5"). InnerVolumeSpecName "kube-api-access-bnmk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:20:23.262401 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.262377    4327 scope.go:122] "RemoveContainer" containerID="e90f60efab88ef607254881b9b829967130a5a8c64e46ace7fb6dcbb0eabdc6f"
Oct 31 01:20:23.265423 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:20:23.265387    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e90f60efab88ef607254881b9b829967130a5a8c64e46ace7fb6dcbb0eabdc6f\": container with ID starting with e90f60efab88ef607254881b9b829967130a5a8c64e46ace7fb6dcbb0eabdc6f not found: ID does not exist" containerID="e90f60efab88ef607254881b9b829967130a5a8c64e46ace7fb6dcbb0eabdc6f"
Oct 31 01:20:23.265564 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.265426    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e90f60efab88ef607254881b9b829967130a5a8c64e46ace7fb6dcbb0eabdc6f"} err="failed to get container status \"e90f60efab88ef607254881b9b829967130a5a8c64e46ace7fb6dcbb0eabdc6f\": rpc error: code = NotFound desc = could not find container \"e90f60efab88ef607254881b9b829967130a5a8c64e46ace7fb6dcbb0eabdc6f\": container with ID starting with e90f60efab88ef607254881b9b829967130a5a8c64e46ace7fb6dcbb0eabdc6f not found: ID does not exist"
Oct 31 01:20:23.294067 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.294016    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk"]
Oct 31 01:20:23.298928 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.298896    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-3c90d8-5dbcb4889c-vd2hk"]
Oct 31 01:20:23.361411 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.361367    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b302fecf-fc39-4122-bc35-83585cc68ae5-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:23.361411 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.361405    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnmk2\" (UniqueName: \"kubernetes.io/projected/b302fecf-fc39-4122-bc35-83585cc68ae5-kube-api-access-bnmk2\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:23.589147 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.589114    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt"]
Oct 31 01:20:23.593296 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.593265    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3c90d8-b9cc8fff-nm2dt"]
Oct 31 01:20:23.923619 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.923578    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67146ff0-b14c-4bb9-9a6b-db437c984bbd" path="/var/lib/kubelet/pods/67146ff0-b14c-4bb9-9a6b-db437c984bbd/volumes"
Oct 31 01:20:23.924098 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:23.924080    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b302fecf-fc39-4122-bc35-83585cc68ae5" path="/var/lib/kubelet/pods/b302fecf-fc39-4122-bc35-83585cc68ae5/volumes"
Oct 31 01:20:25.759343 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:25.759308    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"]
Oct 31 01:20:25.759772 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:25.759542    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz" podUID="5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc" containerName="istio-proxy" containerID="cri-o://aef30089f4d7c883445bcc72f9666b248762e41553a9aacb13680aa0eb94b0f9" gracePeriod=30
Oct 31 01:20:31.046502 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.046437    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:20:31.169174 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.169131    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-token\" (UniqueName: \"kubernetes.io/projected/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-token\") pod \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") "
Oct 31 01:20:31.169389 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.169195    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-workload-socket\") pod \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") "
Oct 31 01:20:31.169389 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.169276    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-kube-api-access-c92jd\" (UniqueName: \"kubernetes.io/projected/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-kube-api-access-c92jd\") pod \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") "
Oct 31 01:20:31.169389 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.169315    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-envoy\") pod \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") "
Oct 31 01:20:31.169389 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.169354    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-data\") pod \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") "
Oct 31 01:20:31.169621 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.169416    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istiod-ca-cert\") pod \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") "
Oct 31 01:20:31.169621 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.169455    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-podinfo\") pod \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") "
Oct 31 01:20:31.169621 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.169529    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-workload-socket" pod "5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc" (UID: "5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:31.169621 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.169551    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-workload-certs\") pod \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") "
Oct 31 01:20:31.169877 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.169631    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-credential-socket\") pod \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\" (UID: \"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc\") "
Oct 31 01:20:31.169973 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.169931    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istiod-ca-cert" pod "5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc" (UID: "5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:20:31.170213 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.170180    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-credential-socket" pod "5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc" (UID: "5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:31.170432 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.170387    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-workload-certs" pod "5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc" (UID: "5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:31.170432 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.170402    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:31.170432 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.170431    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:31.170715 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.170447    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:31.170883 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.170846    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-data" pod "5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc" (UID: "5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:31.172107 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.172085    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-envoy" pod "5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc" (UID: "5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:31.172498 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.172476    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-podinfo" pod "5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc" (UID: "5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:20:31.172793 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.172774    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-token" pod "5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc" (UID: "5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:20:31.173006 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.172988    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-kube-api-access-c92jd" pod "5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc" (UID: "5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc"). InnerVolumeSpecName "kube-api-access-c92jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:20:31.247448 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.247402    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"]
Oct 31 01:20:31.272383 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.272349    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c92jd\" (UniqueName: \"kubernetes.io/projected/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-kube-api-access-c92jd\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:31.272383 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.272380    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:31.272383 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.272391    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:31.272618 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.272400    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:31.272618 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.272408    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:31.272618 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.272417    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:31.303889 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.303792    4327 generic.go:356] "Generic (PLEG): container finished" podID="5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc" containerID="aef30089f4d7c883445bcc72f9666b248762e41553a9aacb13680aa0eb94b0f9" exitCode=0
Oct 31 01:20:31.304072 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.303891    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz" event={"ID":"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc","Type":"ContainerDied","Data":"aef30089f4d7c883445bcc72f9666b248762e41553a9aacb13680aa0eb94b0f9"}
Oct 31 01:20:31.304072 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.303984    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz" event={"ID":"5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc","Type":"ContainerDied","Data":"4a0ec045e984c1ed5cd7af8fe33c865e1e2599140eb9b58d77b9a6e2f0a90cfa"}
Oct 31 01:20:31.304072 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.304024    4327 scope.go:122] "RemoveContainer" containerID="aef30089f4d7c883445bcc72f9666b248762e41553a9aacb13680aa0eb94b0f9"
Oct 31 01:20:31.304473 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.304448    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"
Oct 31 01:20:31.320230 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.320203    4327 scope.go:122] "RemoveContainer" containerID="aef30089f4d7c883445bcc72f9666b248762e41553a9aacb13680aa0eb94b0f9"
Oct 31 01:20:31.320663 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:20:31.320613    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aef30089f4d7c883445bcc72f9666b248762e41553a9aacb13680aa0eb94b0f9\": container with ID starting with aef30089f4d7c883445bcc72f9666b248762e41553a9aacb13680aa0eb94b0f9 not found: ID does not exist" containerID="aef30089f4d7c883445bcc72f9666b248762e41553a9aacb13680aa0eb94b0f9"
Oct 31 01:20:31.320827 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.320687    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aef30089f4d7c883445bcc72f9666b248762e41553a9aacb13680aa0eb94b0f9"} err="failed to get container status \"aef30089f4d7c883445bcc72f9666b248762e41553a9aacb13680aa0eb94b0f9\": rpc error: code = NotFound desc = could not find container \"aef30089f4d7c883445bcc72f9666b248762e41553a9aacb13680aa0eb94b0f9\": container with ID starting with aef30089f4d7c883445bcc72f9666b248762e41553a9aacb13680aa0eb94b0f9 not found: ID does not exist"
Oct 31 01:20:31.362526 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.362491    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"]
Oct 31 01:20:31.371674 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.371647    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-gqkcz"]
Oct 31 01:20:31.922029 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:31.921994    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc" path="/var/lib/kubelet/pods/5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc/volumes"
Oct 31 01:20:42.718059 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:42.718016    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"]
Oct 31 01:20:42.720729 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:42.718239    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s" podUID="c6879302-3938-426c-a79b-8738dee89964" containerName="istio-proxy" containerID="cri-o://9a463c5dfb650be76fdde49ef85419e7cda82b7de583521b94d28ecd2e4a57f4" gracePeriod=30
Oct 31 01:20:44.877459 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.877424    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k"]
Oct 31 01:20:44.878301 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878270    4327 cpu_manager.go:414] "containerMap: removing container" podUID="55c787a6-2c98-4474-ac32-82f652780c14" containerName="maas-api"
Oct 31 01:20:44.878417 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878304    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55c787a6-2c98-4474-ac32-82f652780c14" containerName="maas-api"
Oct 31 01:20:44.878417 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878330    4327 cpu_manager.go:414] "containerMap: removing container" podUID="cae467a1-6a9a-4c1b-855a-ac8edb1d10ee" containerName="istio-proxy"
Oct 31 01:20:44.878417 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878340    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cae467a1-6a9a-4c1b-855a-ac8edb1d10ee" containerName="istio-proxy"
Oct 31 01:20:44.878417 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878355    4327 cpu_manager.go:414] "containerMap: removing container" podUID="d965e342-4052-4f57-a495-4d8023a1f882" containerName="istio-proxy"
Oct 31 01:20:44.878417 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878364    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d965e342-4052-4f57-a495-4d8023a1f882" containerName="istio-proxy"
Oct 31 01:20:44.878417 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878374    4327 cpu_manager.go:414] "containerMap: removing container" podUID="4c28e892-156c-4df2-a55e-af21750b4fda" containerName="istio-proxy"
Oct 31 01:20:44.878417 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878381    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c28e892-156c-4df2-a55e-af21750b4fda" containerName="istio-proxy"
Oct 31 01:20:44.878417 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878397    4327 cpu_manager.go:414] "containerMap: removing container" podUID="9949e8e5-8f11-4c06-8a7b-20e8640c2ac6" containerName="payload-pre-processing"
Oct 31 01:20:44.878417 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878405    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9949e8e5-8f11-4c06-8a7b-20e8640c2ac6" containerName="payload-pre-processing"
Oct 31 01:20:44.878787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878437    4327 cpu_manager.go:414] "containerMap: removing container" podUID="d965e342-4052-4f57-a495-4d8023a1f882" containerName="istio-proxy"
Oct 31 01:20:44.878787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878444    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d965e342-4052-4f57-a495-4d8023a1f882" containerName="istio-proxy"
Oct 31 01:20:44.878787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878455    4327 cpu_manager.go:414] "containerMap: removing container" podUID="5fe10361-91bc-4ff6-a6d4-faec973ad665" containerName="payload-processing"
Oct 31 01:20:44.878787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878461    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fe10361-91bc-4ff6-a6d4-faec973ad665" containerName="payload-processing"
Oct 31 01:20:44.878787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878483    4327 cpu_manager.go:414] "containerMap: removing container" podUID="96e41c65-75e1-4501-b965-14fa687c68c9" containerName="payload-processing"
Oct 31 01:20:44.878787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878492    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e41c65-75e1-4501-b965-14fa687c68c9" containerName="payload-processing"
Oct 31 01:20:44.878787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878503    4327 cpu_manager.go:414] "containerMap: removing container" podUID="b302fecf-fc39-4122-bc35-83585cc68ae5" containerName="payload-pre-processing"
Oct 31 01:20:44.878787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878510    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b302fecf-fc39-4122-bc35-83585cc68ae5" containerName="payload-pre-processing"
Oct 31 01:20:44.878787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878526    4327 cpu_manager.go:414] "containerMap: removing container" podUID="852c332c-c02b-4ccc-b4ef-8733645922c9" containerName="payload-pre-processing"
Oct 31 01:20:44.878787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878534    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="852c332c-c02b-4ccc-b4ef-8733645922c9" containerName="payload-pre-processing"
Oct 31 01:20:44.878787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878557    4327 cpu_manager.go:414] "containerMap: removing container" podUID="5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc" containerName="istio-proxy"
Oct 31 01:20:44.878787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878564    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc" containerName="istio-proxy"
Oct 31 01:20:44.878787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878576    4327 cpu_manager.go:414] "containerMap: removing container" podUID="67146ff0-b14c-4bb9-9a6b-db437c984bbd" containerName="payload-processing"
Oct 31 01:20:44.878787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878583    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67146ff0-b14c-4bb9-9a6b-db437c984bbd" containerName="payload-processing"
Oct 31 01:20:44.878787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878607    4327 cpu_manager.go:414] "containerMap: removing container" podUID="e576a213-4352-4cea-b68a-388edabf4ab0" containerName="payload-pre-processing"
Oct 31 01:20:44.878787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878614    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e576a213-4352-4cea-b68a-388edabf4ab0" containerName="payload-pre-processing"
Oct 31 01:20:44.878787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878630    4327 cpu_manager.go:414] "containerMap: removing container" podUID="2c2c3784-1773-40a2-b373-f925c7e4d239" containerName="payload-processing"
Oct 31 01:20:44.878787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878637    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c2c3784-1773-40a2-b373-f925c7e4d239" containerName="payload-processing"
Oct 31 01:20:44.879541 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878820    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c2c3784-1773-40a2-b373-f925c7e4d239" containerName="payload-processing"
Oct 31 01:20:44.879541 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878833    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="55c787a6-2c98-4474-ac32-82f652780c14" containerName="maas-api"
Oct 31 01:20:44.879541 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878840    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="d965e342-4052-4f57-a495-4d8023a1f882" containerName="istio-proxy"
Oct 31 01:20:44.879541 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878847    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c28e892-156c-4df2-a55e-af21750b4fda" containerName="istio-proxy"
Oct 31 01:20:44.879541 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878854    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="9949e8e5-8f11-4c06-8a7b-20e8640c2ac6" containerName="payload-pre-processing"
Oct 31 01:20:44.879541 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878863    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="13ff4194-5ef3-4f4d-b6e7-4cb5df757371" containerName="istio-proxy"
Oct 31 01:20:44.879541 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878870    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fe10361-91bc-4ff6-a6d4-faec973ad665" containerName="payload-processing"
Oct 31 01:20:44.879541 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878878    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e41c65-75e1-4501-b965-14fa687c68c9" containerName="payload-processing"
Oct 31 01:20:44.879541 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878883    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="b302fecf-fc39-4122-bc35-83585cc68ae5" containerName="payload-pre-processing"
Oct 31 01:20:44.879541 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878889    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="852c332c-c02b-4ccc-b4ef-8733645922c9" containerName="payload-pre-processing"
Oct 31 01:20:44.879541 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878898    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d8d7b3f-f478-4b30-b9be-6b9b86ff1fbc" containerName="istio-proxy"
Oct 31 01:20:44.879541 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878904    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="67146ff0-b14c-4bb9-9a6b-db437c984bbd" containerName="payload-processing"
Oct 31 01:20:44.879541 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.878911    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="e576a213-4352-4cea-b68a-388edabf4ab0" containerName="payload-pre-processing"
Oct 31 01:20:44.883226 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.883198    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k"
Oct 31 01:20:44.906269 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.906232    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k"]
Oct 31 01:20:44.914531 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.914496    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-114c95\""
Oct 31 01:20:44.914791 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:44.914696    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-114c95-dockercfg-kbrq8\""
Oct 31 01:20:45.044559 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:45.044522    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g"]
Oct 31 01:20:45.045340 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:45.045315    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="d965e342-4052-4f57-a495-4d8023a1f882" containerName="istio-proxy"
Oct 31 01:20:45.048934 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:45.048915    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g"
Oct 31 01:20:45.057847 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:45.057819    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp4wd\" (UniqueName: \"kubernetes.io/projected/edf4f19d-74fd-4f83-96d2-6c66d711fa24-kube-api-access-jp4wd\") pod \"payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k\" (UID: \"edf4f19d-74fd-4f83-96d2-6c66d711fa24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k"
Oct 31 01:20:45.058006 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:45.057928    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edf4f19d-74fd-4f83-96d2-6c66d711fa24-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k\" (UID: \"edf4f19d-74fd-4f83-96d2-6c66d711fa24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k"
Oct 31 01:20:45.083420 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:45.071265    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g"]
Oct 31 01:20:45.159912 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:45.159802    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp4wd\" (UniqueName: \"kubernetes.io/projected/edf4f19d-74fd-4f83-96d2-6c66d711fa24-kube-api-access-jp4wd\") pod \"payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k\" (UID: \"edf4f19d-74fd-4f83-96d2-6c66d711fa24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k"
Oct 31 01:20:45.160102 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:45.159950    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edf4f19d-74fd-4f83-96d2-6c66d711fa24-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k\" (UID: \"edf4f19d-74fd-4f83-96d2-6c66d711fa24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k"
Oct 31 01:20:45.160153 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:45.160122    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9flcx\" (UniqueName: \"kubernetes.io/projected/ceab7625-0443-4896-a735-5e917167771a-kube-api-access-9flcx\") pod \"payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g\" (UID: \"ceab7625-0443-4896-a735-5e917167771a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g"
Oct 31 01:20:45.160289 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:45.160265    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceab7625-0443-4896-a735-5e917167771a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g\" (UID: \"ceab7625-0443-4896-a735-5e917167771a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g"
Oct 31 01:20:45.160695 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:45.160664    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edf4f19d-74fd-4f83-96d2-6c66d711fa24-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k\" (UID: \"edf4f19d-74fd-4f83-96d2-6c66d711fa24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k"
Oct 31 01:20:45.183855 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:45.183815    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp4wd\" (UniqueName: \"kubernetes.io/projected/edf4f19d-74fd-4f83-96d2-6c66d711fa24-kube-api-access-jp4wd\") pod \"payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k\" (UID: \"edf4f19d-74fd-4f83-96d2-6c66d711fa24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k"
Oct 31 01:20:45.199133 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:45.199108    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k"
Oct 31 01:20:45.261597 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:45.261558    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9flcx\" (UniqueName: \"kubernetes.io/projected/ceab7625-0443-4896-a735-5e917167771a-kube-api-access-9flcx\") pod \"payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g\" (UID: \"ceab7625-0443-4896-a735-5e917167771a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g"
Oct 31 01:20:45.261931 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:45.261906    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceab7625-0443-4896-a735-5e917167771a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g\" (UID: \"ceab7625-0443-4896-a735-5e917167771a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g"
Oct 31 01:20:45.262823 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:45.262774    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceab7625-0443-4896-a735-5e917167771a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g\" (UID: \"ceab7625-0443-4896-a735-5e917167771a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g"
Oct 31 01:20:45.285621 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:45.285589    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9flcx\" (UniqueName: \"kubernetes.io/projected/ceab7625-0443-4896-a735-5e917167771a-kube-api-access-9flcx\") pod \"payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g\" (UID: \"ceab7625-0443-4896-a735-5e917167771a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g"
Oct 31 01:20:45.301039 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:20:45.301001    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:20:45.366251 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:45.366186    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g"
Oct 31 01:20:45.501035 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:45.501001    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k"]
Oct 31 01:20:45.501701 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:20:45.501657    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf4f19d_74fd_4f83_96d2_6c66d711fa24.slice/crio-62add354ee07531f379ed3a76c5e8ee651f6c7212699a2ed4ab9db80a5be5ab1 WatchSource:0}: Error finding container 62add354ee07531f379ed3a76c5e8ee651f6c7212699a2ed4ab9db80a5be5ab1: Status 404 returned error can't find the container with id 62add354ee07531f379ed3a76c5e8ee651f6c7212699a2ed4ab9db80a5be5ab1
Oct 31 01:20:45.674018 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:45.673987    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g"]
Oct 31 01:20:45.674163 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:20:45.674124    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceab7625_0443_4896_a735_5e917167771a.slice/crio-d766eecf7bdc86e9ef42b44fd1549573b4fa1206c193683e80d9bf40e43cf50a WatchSource:0}: Error finding container d766eecf7bdc86e9ef42b44fd1549573b4fa1206c193683e80d9bf40e43cf50a: Status 404 returned error can't find the container with id d766eecf7bdc86e9ef42b44fd1549573b4fa1206c193683e80d9bf40e43cf50a
Oct 31 01:20:46.272413 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.272367    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" containerID="cri-o://1d9f01421ca4002b44bf489e9ed02ef068d5ab583d9036cb075ac478a0323b48" gracePeriod=45
Oct 31 01:20:46.397146 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.397108    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": dial tcp 10.133.0.66:8000: connect: connection refused"
Oct 31 01:20:46.417230 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.417182    4327 generic.go:356] "Generic (PLEG): container finished" podID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerID="1d9f01421ca4002b44bf489e9ed02ef068d5ab583d9036cb075ac478a0323b48" exitCode=0
Oct 31 01:20:46.417425 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.417265    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" event={"ID":"52f30ed7-5edf-403e-b75c-53f8b27f94bd","Type":"ContainerDied","Data":"1d9f01421ca4002b44bf489e9ed02ef068d5ab583d9036cb075ac478a0323b48"}
Oct 31 01:20:46.419063 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.419035    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g" event={"ID":"ceab7625-0443-4896-a735-5e917167771a","Type":"ContainerStarted","Data":"624c37fbb148f3926a087d15392e4b6e3f0b14c956f58b1916f9c093dce70d8d"}
Oct 31 01:20:46.420231 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.420207    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g"
Oct 31 01:20:46.420498 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.420478    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g" event={"ID":"ceab7625-0443-4896-a735-5e917167771a","Type":"ContainerStarted","Data":"d766eecf7bdc86e9ef42b44fd1549573b4fa1206c193683e80d9bf40e43cf50a"}
Oct 31 01:20:46.420910 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.420887    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k" event={"ID":"edf4f19d-74fd-4f83-96d2-6c66d711fa24","Type":"ContainerStarted","Data":"54ca17c793ec754eb9d63059cf6fc1879105de9643defc7d2b65fbdd5616818a"}
Oct 31 01:20:46.420981 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.420929    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k" event={"ID":"edf4f19d-74fd-4f83-96d2-6c66d711fa24","Type":"ContainerStarted","Data":"62add354ee07531f379ed3a76c5e8ee651f6c7212699a2ed4ab9db80a5be5ab1"}
Oct 31 01:20:46.420981 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.420969    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k"
Oct 31 01:20:46.466264 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.466211    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g" podStartSLOduration=1.4661962210000001 podStartE2EDuration="1.466196221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:46.461901508 +0000 UTC m=+3706.683937063" watchObservedRunningTime="2026-10-31 01:20:46.466196221 +0000 UTC m=+3706.688231762"
Oct 31 01:20:46.507305 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.507240    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k" podStartSLOduration=2.507220562 podStartE2EDuration="2.507220562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:46.506106421 +0000 UTC m=+3706.728141962" watchObservedRunningTime="2026-10-31 01:20:46.507220562 +0000 UTC m=+3706.729256106"
Oct 31 01:20:46.563968 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.563937    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 01:20:46.682545 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.682506    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-model-cache\") pod \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\" (UID: \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\") "
Oct 31 01:20:46.682774 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.682553    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-tmp-dir\") pod \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\" (UID: \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\") "
Oct 31 01:20:46.682854 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.682811    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52f30ed7-5edf-403e-b75c-53f8b27f94bd-tls-certs\" (UniqueName: \"kubernetes.io/secret/52f30ed7-5edf-403e-b75c-53f8b27f94bd-tls-certs\") pod \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\" (UID: \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\") "
Oct 31 01:20:46.682927 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.682848    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-model-cache" pod "52f30ed7-5edf-403e-b75c-53f8b27f94bd" (UID: "52f30ed7-5edf-403e-b75c-53f8b27f94bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:46.682927 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.682864    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-home\" (UniqueName: \"kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-home\") pod \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\" (UID: \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\") "
Oct 31 01:20:46.683048 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.682941    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-dshm\") pod \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\" (UID: \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\") "
Oct 31 01:20:46.683108 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.683080    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52f30ed7-5edf-403e-b75c-53f8b27f94bd-kube-api-access-w7klf\" (UniqueName: \"kubernetes.io/projected/52f30ed7-5edf-403e-b75c-53f8b27f94bd-kube-api-access-w7klf\") pod \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\" (UID: \"52f30ed7-5edf-403e-b75c-53f8b27f94bd\") "
Oct 31 01:20:46.683108 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.683094    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-home" pod "52f30ed7-5edf-403e-b75c-53f8b27f94bd" (UID: "52f30ed7-5edf-403e-b75c-53f8b27f94bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:46.683310 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.683287    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-tmp-dir" pod "52f30ed7-5edf-403e-b75c-53f8b27f94bd" (UID: "52f30ed7-5edf-403e-b75c-53f8b27f94bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:46.683625 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.683602    4327 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-model-cache\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:46.683625 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.683624    4327 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-tmp-dir\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:46.683808 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.683637    4327 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-home\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:46.686921 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.686894    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f30ed7-5edf-403e-b75c-53f8b27f94bd-kube-api-access-w7klf" pod "52f30ed7-5edf-403e-b75c-53f8b27f94bd" (UID: "52f30ed7-5edf-403e-b75c-53f8b27f94bd"). InnerVolumeSpecName "kube-api-access-w7klf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:20:46.687135 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.687108    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52f30ed7-5edf-403e-b75c-53f8b27f94bd-tls-certs" pod "52f30ed7-5edf-403e-b75c-53f8b27f94bd" (UID: "52f30ed7-5edf-403e-b75c-53f8b27f94bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:20:46.692245 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.692215    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-dshm" pod "52f30ed7-5edf-403e-b75c-53f8b27f94bd" (UID: "52f30ed7-5edf-403e-b75c-53f8b27f94bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:46.785453 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.785368    4327 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f30ed7-5edf-403e-b75c-53f8b27f94bd-tls-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:46.785453 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.785401    4327 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f30ed7-5edf-403e-b75c-53f8b27f94bd-dshm\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:46.785453 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:46.785414    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7klf\" (UniqueName: \"kubernetes.io/projected/52f30ed7-5edf-403e-b75c-53f8b27f94bd-kube-api-access-w7klf\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:47.429604 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:47.429572    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"
Oct 31 01:20:47.430135 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:47.429602    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs" event={"ID":"52f30ed7-5edf-403e-b75c-53f8b27f94bd","Type":"ContainerDied","Data":"c8b5d983abc2117f6da897f45e23e9c5b4bceebabdf8bf1b239fafd112aaf012"}
Oct 31 01:20:47.430135 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:47.429661    4327 scope.go:122] "RemoveContainer" containerID="1d9f01421ca4002b44bf489e9ed02ef068d5ab583d9036cb075ac478a0323b48"
Oct 31 01:20:47.483305 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:47.483262    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"]
Oct 31 01:20:47.495715 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:47.495669    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-671959/distinct-2-w0-671959-kserve-7dccd77d97-q8wzs"]
Oct 31 01:20:47.923269 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:47.923232    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" path="/var/lib/kubelet/pods/52f30ed7-5edf-403e-b75c-53f8b27f94bd/volumes"
Oct 31 01:20:47.999759 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:47.999695    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:20:48.106454 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.106414    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-istio-envoy\") pod \"c6879302-3938-426c-a79b-8738dee89964\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") "
Oct 31 01:20:48.106634 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.106497    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-credential-socket\") pod \"c6879302-3938-426c-a79b-8738dee89964\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") "
Oct 31 01:20:48.106634 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.106524    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6879302-3938-426c-a79b-8738dee89964-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6879302-3938-426c-a79b-8738dee89964-istiod-ca-cert\") pod \"c6879302-3938-426c-a79b-8738dee89964\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") "
Oct 31 01:20:48.106634 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.106566    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6879302-3938-426c-a79b-8738dee89964-kube-api-access-dhqtn\" (UniqueName: \"kubernetes.io/projected/c6879302-3938-426c-a79b-8738dee89964-kube-api-access-dhqtn\") pod \"c6879302-3938-426c-a79b-8738dee89964\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") "
Oct 31 01:20:48.106810 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.106651    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-workload-certs\") pod \"c6879302-3938-426c-a79b-8738dee89964\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") "
Oct 31 01:20:48.106810 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.106711    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6879302-3938-426c-a79b-8738dee89964-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6879302-3938-426c-a79b-8738dee89964-istio-podinfo\") pod \"c6879302-3938-426c-a79b-8738dee89964\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") "
Oct 31 01:20:48.106810 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.106765    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-workload-socket\") pod \"c6879302-3938-426c-a79b-8738dee89964\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") "
Oct 31 01:20:48.106970 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.106808    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6879302-3938-426c-a79b-8738dee89964-istio-token\" (UniqueName: \"kubernetes.io/projected/c6879302-3938-426c-a79b-8738dee89964-istio-token\") pod \"c6879302-3938-426c-a79b-8738dee89964\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") "
Oct 31 01:20:48.106970 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.106835    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-istio-data\") pod \"c6879302-3938-426c-a79b-8738dee89964\" (UID: \"c6879302-3938-426c-a79b-8738dee89964\") "
Oct 31 01:20:48.107097 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.106994    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-workload-certs" pod "c6879302-3938-426c-a79b-8738dee89964" (UID: "c6879302-3938-426c-a79b-8738dee89964"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:48.107097 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.107036    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-workload-socket" pod "c6879302-3938-426c-a79b-8738dee89964" (UID: "c6879302-3938-426c-a79b-8738dee89964"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:48.107215 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.107087    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6879302-3938-426c-a79b-8738dee89964-istiod-ca-cert" pod "c6879302-3938-426c-a79b-8738dee89964" (UID: "c6879302-3938-426c-a79b-8738dee89964"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:20:48.107215 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.107161    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-credential-socket" pod "c6879302-3938-426c-a79b-8738dee89964" (UID: "c6879302-3938-426c-a79b-8738dee89964"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:48.107332 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.107235    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-istio-data" pod "c6879302-3938-426c-a79b-8738dee89964" (UID: "c6879302-3938-426c-a79b-8738dee89964"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:48.107548 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.107522    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:48.107643 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.107552    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6879302-3938-426c-a79b-8738dee89964-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:48.107643 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.107567    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:48.107643 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.107582    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:48.107643 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.107597    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:48.109053 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.109026    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-istio-envoy" pod "c6879302-3938-426c-a79b-8738dee89964" (UID: "c6879302-3938-426c-a79b-8738dee89964"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:48.109053 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.109028    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6879302-3938-426c-a79b-8738dee89964-istio-podinfo" pod "c6879302-3938-426c-a79b-8738dee89964" (UID: "c6879302-3938-426c-a79b-8738dee89964"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:20:48.109208 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.109033    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6879302-3938-426c-a79b-8738dee89964-kube-api-access-dhqtn" pod "c6879302-3938-426c-a79b-8738dee89964" (UID: "c6879302-3938-426c-a79b-8738dee89964"). InnerVolumeSpecName "kube-api-access-dhqtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:20:48.109208 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.109154    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6879302-3938-426c-a79b-8738dee89964-istio-token" pod "c6879302-3938-426c-a79b-8738dee89964" (UID: "c6879302-3938-426c-a79b-8738dee89964"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:20:48.208535 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.208502    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6879302-3938-426c-a79b-8738dee89964-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:48.208535 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.208529    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6879302-3938-426c-a79b-8738dee89964-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:48.208535 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.208539    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6879302-3938-426c-a79b-8738dee89964-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:48.208811 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.208547    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhqtn\" (UniqueName: \"kubernetes.io/projected/c6879302-3938-426c-a79b-8738dee89964-kube-api-access-dhqtn\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:48.440650 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.440608    4327 generic.go:356] "Generic (PLEG): container finished" podID="c6879302-3938-426c-a79b-8738dee89964" containerID="9a463c5dfb650be76fdde49ef85419e7cda82b7de583521b94d28ecd2e4a57f4" exitCode=0
Oct 31 01:20:48.441184 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.440673    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"
Oct 31 01:20:48.441184 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.440735    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s" event={"ID":"c6879302-3938-426c-a79b-8738dee89964","Type":"ContainerDied","Data":"9a463c5dfb650be76fdde49ef85419e7cda82b7de583521b94d28ecd2e4a57f4"}
Oct 31 01:20:48.441184 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.440823    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s" event={"ID":"c6879302-3938-426c-a79b-8738dee89964","Type":"ContainerDied","Data":"5b17498c45b277f7a508b5e611aa4d20f1f65ca626d676d7918789a0863d7618"}
Oct 31 01:20:48.441184 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.440883    4327 scope.go:122] "RemoveContainer" containerID="9a463c5dfb650be76fdde49ef85419e7cda82b7de583521b94d28ecd2e4a57f4"
Oct 31 01:20:48.461609 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.461586    4327 scope.go:122] "RemoveContainer" containerID="9a463c5dfb650be76fdde49ef85419e7cda82b7de583521b94d28ecd2e4a57f4"
Oct 31 01:20:48.461998 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:20:48.461942    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a463c5dfb650be76fdde49ef85419e7cda82b7de583521b94d28ecd2e4a57f4\": container with ID starting with 9a463c5dfb650be76fdde49ef85419e7cda82b7de583521b94d28ecd2e4a57f4 not found: ID does not exist" containerID="9a463c5dfb650be76fdde49ef85419e7cda82b7de583521b94d28ecd2e4a57f4"
Oct 31 01:20:48.461998 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.461983    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a463c5dfb650be76fdde49ef85419e7cda82b7de583521b94d28ecd2e4a57f4"} err="failed to get container status \"9a463c5dfb650be76fdde49ef85419e7cda82b7de583521b94d28ecd2e4a57f4\": rpc error: code = NotFound desc = could not find container \"9a463c5dfb650be76fdde49ef85419e7cda82b7de583521b94d28ecd2e4a57f4\": container with ID starting with 9a463c5dfb650be76fdde49ef85419e7cda82b7de583521b94d28ecd2e4a57f4 not found: ID does not exist"
Oct 31 01:20:48.496322 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.496296    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"]
Oct 31 01:20:48.508878 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:48.508846    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b674s"]
Oct 31 01:20:49.923391 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:49.923359    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6879302-3938-426c-a79b-8738dee89964" path="/var/lib/kubelet/pods/c6879302-3938-426c-a79b-8738dee89964/volumes"
Oct 31 01:20:50.564859 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:50.564824    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk"]
Oct 31 01:20:50.565089 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:50.565063    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk" podUID="b6a22f7c-91e0-44af-bcf7-f987701bc859" containerName="maas-api" containerID="cri-o://5c0cfbe8debebacd96c934ead2a56f0e020b07b04c96ba5699622b48c4328d24" gracePeriod=30
Oct 31 01:20:50.877104 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:50.877073    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk"
Oct 31 01:20:50.942347 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:50.942292    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6a22f7c-91e0-44af-bcf7-f987701bc859-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6a22f7c-91e0-44af-bcf7-f987701bc859-metrics-certs\") pod \"b6a22f7c-91e0-44af-bcf7-f987701bc859\" (UID: \"b6a22f7c-91e0-44af-bcf7-f987701bc859\") "
Oct 31 01:20:50.942883 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:50.942361    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6a22f7c-91e0-44af-bcf7-f987701bc859-kube-api-access-tqd94\" (UniqueName: \"kubernetes.io/projected/b6a22f7c-91e0-44af-bcf7-f987701bc859-kube-api-access-tqd94\") pod \"b6a22f7c-91e0-44af-bcf7-f987701bc859\" (UID: \"b6a22f7c-91e0-44af-bcf7-f987701bc859\") "
Oct 31 01:20:50.942883 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:50.942460    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6a22f7c-91e0-44af-bcf7-f987701bc859-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a22f7c-91e0-44af-bcf7-f987701bc859-openshift-service-ca\") pod \"b6a22f7c-91e0-44af-bcf7-f987701bc859\" (UID: \"b6a22f7c-91e0-44af-bcf7-f987701bc859\") "
Oct 31 01:20:50.942883 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:50.942509    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6a22f7c-91e0-44af-bcf7-f987701bc859-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6a22f7c-91e0-44af-bcf7-f987701bc859-maas-api-tls\") pod \"b6a22f7c-91e0-44af-bcf7-f987701bc859\" (UID: \"b6a22f7c-91e0-44af-bcf7-f987701bc859\") "
Oct 31 01:20:50.944288 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:50.944260    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6a22f7c-91e0-44af-bcf7-f987701bc859-openshift-service-ca" pod "b6a22f7c-91e0-44af-bcf7-f987701bc859" (UID: "b6a22f7c-91e0-44af-bcf7-f987701bc859"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:20:50.944861 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:50.944836    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6a22f7c-91e0-44af-bcf7-f987701bc859-maas-api-tls" pod "b6a22f7c-91e0-44af-bcf7-f987701bc859" (UID: "b6a22f7c-91e0-44af-bcf7-f987701bc859"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:20:50.945413 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:50.945392    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6a22f7c-91e0-44af-bcf7-f987701bc859-metrics-certs" pod "b6a22f7c-91e0-44af-bcf7-f987701bc859" (UID: "b6a22f7c-91e0-44af-bcf7-f987701bc859"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:20:50.945777 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:50.945739    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a22f7c-91e0-44af-bcf7-f987701bc859-kube-api-access-tqd94" pod "b6a22f7c-91e0-44af-bcf7-f987701bc859" (UID: "b6a22f7c-91e0-44af-bcf7-f987701bc859"). InnerVolumeSpecName "kube-api-access-tqd94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:20:51.044414 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:51.044374    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6a22f7c-91e0-44af-bcf7-f987701bc859-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:51.044414 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:51.044415    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqd94\" (UniqueName: \"kubernetes.io/projected/b6a22f7c-91e0-44af-bcf7-f987701bc859-kube-api-access-tqd94\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:51.044731 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:51.044432    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a22f7c-91e0-44af-bcf7-f987701bc859-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:51.044731 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:51.044446    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6a22f7c-91e0-44af-bcf7-f987701bc859-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:20:51.464466 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:51.464431    4327 generic.go:356] "Generic (PLEG): container finished" podID="b6a22f7c-91e0-44af-bcf7-f987701bc859" containerID="5c0cfbe8debebacd96c934ead2a56f0e020b07b04c96ba5699622b48c4328d24" exitCode=0
Oct 31 01:20:51.464466 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:51.464470    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk" event={"ID":"b6a22f7c-91e0-44af-bcf7-f987701bc859","Type":"ContainerDied","Data":"5c0cfbe8debebacd96c934ead2a56f0e020b07b04c96ba5699622b48c4328d24"}
Oct 31 01:20:51.464795 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:51.464483    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk"
Oct 31 01:20:51.464795 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:51.464517    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk" event={"ID":"b6a22f7c-91e0-44af-bcf7-f987701bc859","Type":"ContainerDied","Data":"570cb20c27b9cbfefba5062f0b9fbd863feb5729a1ba080f7356256af36e1ac7"}
Oct 31 01:20:51.464795 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:51.464541    4327 scope.go:122] "RemoveContainer" containerID="5c0cfbe8debebacd96c934ead2a56f0e020b07b04c96ba5699622b48c4328d24"
Oct 31 01:20:51.475093 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:51.475073    4327 scope.go:122] "RemoveContainer" containerID="5c0cfbe8debebacd96c934ead2a56f0e020b07b04c96ba5699622b48c4328d24"
Oct 31 01:20:51.475357 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:20:51.475338    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c0cfbe8debebacd96c934ead2a56f0e020b07b04c96ba5699622b48c4328d24\": container with ID starting with 5c0cfbe8debebacd96c934ead2a56f0e020b07b04c96ba5699622b48c4328d24 not found: ID does not exist" containerID="5c0cfbe8debebacd96c934ead2a56f0e020b07b04c96ba5699622b48c4328d24"
Oct 31 01:20:51.475408 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:51.475369    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c0cfbe8debebacd96c934ead2a56f0e020b07b04c96ba5699622b48c4328d24"} err="failed to get container status \"5c0cfbe8debebacd96c934ead2a56f0e020b07b04c96ba5699622b48c4328d24\": rpc error: code = NotFound desc = could not find container \"5c0cfbe8debebacd96c934ead2a56f0e020b07b04c96ba5699622b48c4328d24\": container with ID starting with 5c0cfbe8debebacd96c934ead2a56f0e020b07b04c96ba5699622b48c4328d24 not found: ID does not exist"
Oct 31 01:20:51.520906 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:51.520866    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk"]
Oct 31 01:20:51.532925 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:51.532898    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d251f-64bd776f65-llhzk"]
Oct 31 01:20:51.923961 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:51.923925    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6a22f7c-91e0-44af-bcf7-f987701bc859" path="/var/lib/kubelet/pods/b6a22f7c-91e0-44af-bcf7-f987701bc859/volumes"
Oct 31 01:20:57.434311 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:57.434276    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k"
Oct 31 01:20:58.443649 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:20:58.443622    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g"
Oct 31 01:21:10.304770 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:10.304735    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb"]
Oct 31 01:21:13.971850 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:13.971608    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh"]
Oct 31 01:21:13.972342 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:13.971901    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh" podUID="0280fb12-2007-4c5c-89ca-bf2f18dc28ad" containerName="maas-api" containerID="cri-o://40bdab51d128b374918138955f1c30b420938a057ec79ca589171aeb4a6b335f" gracePeriod=30
Oct 31 01:21:14.121090 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.121050    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq"]
Oct 31 01:21:14.121277 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.121256    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq" podUID="9ec0db43-a0c1-4e82-970b-dbe3f5585acd" containerName="payload-processing" containerID="cri-o://8dfacadceaa759c3a87bc15928442bf791a5038b24a6ab8ea4aa0caa5752b286" gracePeriod=30
Oct 31 01:21:14.143348 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.143311    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b"]
Oct 31 01:21:14.143789 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.143588    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b" podUID="54718bb1-f732-4c6b-aa42-7eda6ecec318" containerName="payload-pre-processing" containerID="cri-o://0806ac6bb7767ffb554a440348ce5e3a6a1febab8a8c69709b47e4486f9dd4e9" gracePeriod=30
Oct 31 01:21:14.487738 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.487672    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq"
Oct 31 01:21:14.558017 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.557988    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b"
Oct 31 01:21:14.608307 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.608274    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh"
Oct 31 01:21:14.627929 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.627887    4327 generic.go:356] "Generic (PLEG): container finished" podID="54718bb1-f732-4c6b-aa42-7eda6ecec318" containerID="0806ac6bb7767ffb554a440348ce5e3a6a1febab8a8c69709b47e4486f9dd4e9" exitCode=0
Oct 31 01:21:14.628141 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.627943    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b"
Oct 31 01:21:14.628141 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.628119    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b" event={"ID":"54718bb1-f732-4c6b-aa42-7eda6ecec318","Type":"ContainerDied","Data":"0806ac6bb7767ffb554a440348ce5e3a6a1febab8a8c69709b47e4486f9dd4e9"}
Oct 31 01:21:14.628280 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.628181    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b" event={"ID":"54718bb1-f732-4c6b-aa42-7eda6ecec318","Type":"ContainerDied","Data":"b5c67bdf5b236366b9878b2a704372bf10fb3f562f50ec1734215fb9d4e64bd5"}
Oct 31 01:21:14.628280 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.628215    4327 scope.go:122] "RemoveContainer" containerID="0806ac6bb7767ffb554a440348ce5e3a6a1febab8a8c69709b47e4486f9dd4e9"
Oct 31 01:21:14.630004 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.629824    4327 generic.go:356] "Generic (PLEG): container finished" podID="9ec0db43-a0c1-4e82-970b-dbe3f5585acd" containerID="8dfacadceaa759c3a87bc15928442bf791a5038b24a6ab8ea4aa0caa5752b286" exitCode=0
Oct 31 01:21:14.630147 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.630129    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq"
Oct 31 01:21:14.630491 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.630356    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq" event={"ID":"9ec0db43-a0c1-4e82-970b-dbe3f5585acd","Type":"ContainerDied","Data":"8dfacadceaa759c3a87bc15928442bf791a5038b24a6ab8ea4aa0caa5752b286"}
Oct 31 01:21:14.630491 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.630409    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq" event={"ID":"9ec0db43-a0c1-4e82-970b-dbe3f5585acd","Type":"ContainerDied","Data":"ba2e2259bbed4d0260154c397e0d7a7ab4355b8753c4c61b2b069c30b92e1a3c"}
Oct 31 01:21:14.632896 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.632873    4327 generic.go:356] "Generic (PLEG): container finished" podID="0280fb12-2007-4c5c-89ca-bf2f18dc28ad" containerID="40bdab51d128b374918138955f1c30b420938a057ec79ca589171aeb4a6b335f" exitCode=0
Oct 31 01:21:14.633029 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.633013    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh" event={"ID":"0280fb12-2007-4c5c-89ca-bf2f18dc28ad","Type":"ContainerDied","Data":"40bdab51d128b374918138955f1c30b420938a057ec79ca589171aeb4a6b335f"}
Oct 31 01:21:14.633083 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.633058    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh" event={"ID":"0280fb12-2007-4c5c-89ca-bf2f18dc28ad","Type":"ContainerDied","Data":"544d3f6b205769448adcb7be5f4171264e7039b38423a4167920c34b4311bc22"}
Oct 31 01:21:14.633140 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.633128    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh"
Oct 31 01:21:14.638386 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.638355    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec0db43-a0c1-4e82-970b-dbe3f5585acd-kube-api-access-rsmzc\" (UniqueName: \"kubernetes.io/projected/9ec0db43-a0c1-4e82-970b-dbe3f5585acd-kube-api-access-rsmzc\") pod \"9ec0db43-a0c1-4e82-970b-dbe3f5585acd\" (UID: \"9ec0db43-a0c1-4e82-970b-dbe3f5585acd\") "
Oct 31 01:21:14.638505 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.638469    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54718bb1-f732-4c6b-aa42-7eda6ecec318-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54718bb1-f732-4c6b-aa42-7eda6ecec318-plugins-config-volume\") pod \"54718bb1-f732-4c6b-aa42-7eda6ecec318\" (UID: \"54718bb1-f732-4c6b-aa42-7eda6ecec318\") "
Oct 31 01:21:14.638629 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.638607    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54718bb1-f732-4c6b-aa42-7eda6ecec318-kube-api-access-lgrkr\" (UniqueName: \"kubernetes.io/projected/54718bb1-f732-4c6b-aa42-7eda6ecec318-kube-api-access-lgrkr\") pod \"54718bb1-f732-4c6b-aa42-7eda6ecec318\" (UID: \"54718bb1-f732-4c6b-aa42-7eda6ecec318\") "
Oct 31 01:21:14.639100 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.639054    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ec0db43-a0c1-4e82-970b-dbe3f5585acd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec0db43-a0c1-4e82-970b-dbe3f5585acd-plugins-config-volume\") pod \"9ec0db43-a0c1-4e82-970b-dbe3f5585acd\" (UID: \"9ec0db43-a0c1-4e82-970b-dbe3f5585acd\") "
Oct 31 01:21:14.639961 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.639930    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec0db43-a0c1-4e82-970b-dbe3f5585acd-plugins-config-volume" pod "9ec0db43-a0c1-4e82-970b-dbe3f5585acd" (UID: "9ec0db43-a0c1-4e82-970b-dbe3f5585acd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:21:14.640302 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.640282    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec0db43-a0c1-4e82-970b-dbe3f5585acd-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:14.640458 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.640311    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54718bb1-f732-4c6b-aa42-7eda6ecec318-plugins-config-volume" pod "54718bb1-f732-4c6b-aa42-7eda6ecec318" (UID: "54718bb1-f732-4c6b-aa42-7eda6ecec318"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:21:14.642135 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.642048    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec0db43-a0c1-4e82-970b-dbe3f5585acd-kube-api-access-rsmzc" pod "9ec0db43-a0c1-4e82-970b-dbe3f5585acd" (UID: "9ec0db43-a0c1-4e82-970b-dbe3f5585acd"). InnerVolumeSpecName "kube-api-access-rsmzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:21:14.642216 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.642160    4327 scope.go:122] "RemoveContainer" containerID="0806ac6bb7767ffb554a440348ce5e3a6a1febab8a8c69709b47e4486f9dd4e9"
Oct 31 01:21:14.642648 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:21:14.642624    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0806ac6bb7767ffb554a440348ce5e3a6a1febab8a8c69709b47e4486f9dd4e9\": container with ID starting with 0806ac6bb7767ffb554a440348ce5e3a6a1febab8a8c69709b47e4486f9dd4e9 not found: ID does not exist" containerID="0806ac6bb7767ffb554a440348ce5e3a6a1febab8a8c69709b47e4486f9dd4e9"
Oct 31 01:21:14.642779 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.642662    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0806ac6bb7767ffb554a440348ce5e3a6a1febab8a8c69709b47e4486f9dd4e9"} err="failed to get container status \"0806ac6bb7767ffb554a440348ce5e3a6a1febab8a8c69709b47e4486f9dd4e9\": rpc error: code = NotFound desc = could not find container \"0806ac6bb7767ffb554a440348ce5e3a6a1febab8a8c69709b47e4486f9dd4e9\": container with ID starting with 0806ac6bb7767ffb554a440348ce5e3a6a1febab8a8c69709b47e4486f9dd4e9 not found: ID does not exist"
Oct 31 01:21:14.642779 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.642698    4327 scope.go:122] "RemoveContainer" containerID="8dfacadceaa759c3a87bc15928442bf791a5038b24a6ab8ea4aa0caa5752b286"
Oct 31 01:21:14.644147 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.644117    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54718bb1-f732-4c6b-aa42-7eda6ecec318-kube-api-access-lgrkr" pod "54718bb1-f732-4c6b-aa42-7eda6ecec318" (UID: "54718bb1-f732-4c6b-aa42-7eda6ecec318"). InnerVolumeSpecName "kube-api-access-lgrkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:21:14.655895 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.655866    4327 scope.go:122] "RemoveContainer" containerID="8dfacadceaa759c3a87bc15928442bf791a5038b24a6ab8ea4aa0caa5752b286"
Oct 31 01:21:14.656272 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:21:14.656245    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dfacadceaa759c3a87bc15928442bf791a5038b24a6ab8ea4aa0caa5752b286\": container with ID starting with 8dfacadceaa759c3a87bc15928442bf791a5038b24a6ab8ea4aa0caa5752b286 not found: ID does not exist" containerID="8dfacadceaa759c3a87bc15928442bf791a5038b24a6ab8ea4aa0caa5752b286"
Oct 31 01:21:14.656347 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.656280    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dfacadceaa759c3a87bc15928442bf791a5038b24a6ab8ea4aa0caa5752b286"} err="failed to get container status \"8dfacadceaa759c3a87bc15928442bf791a5038b24a6ab8ea4aa0caa5752b286\": rpc error: code = NotFound desc = could not find container \"8dfacadceaa759c3a87bc15928442bf791a5038b24a6ab8ea4aa0caa5752b286\": container with ID starting with 8dfacadceaa759c3a87bc15928442bf791a5038b24a6ab8ea4aa0caa5752b286 not found: ID does not exist"
Oct 31 01:21:14.656347 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.656297    4327 scope.go:122] "RemoveContainer" containerID="40bdab51d128b374918138955f1c30b420938a057ec79ca589171aeb4a6b335f"
Oct 31 01:21:14.666626 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.666598    4327 scope.go:122] "RemoveContainer" containerID="40bdab51d128b374918138955f1c30b420938a057ec79ca589171aeb4a6b335f"
Oct 31 01:21:14.667075 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:21:14.667045    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40bdab51d128b374918138955f1c30b420938a057ec79ca589171aeb4a6b335f\": container with ID starting with 40bdab51d128b374918138955f1c30b420938a057ec79ca589171aeb4a6b335f not found: ID does not exist" containerID="40bdab51d128b374918138955f1c30b420938a057ec79ca589171aeb4a6b335f"
Oct 31 01:21:14.667195 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.667081    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40bdab51d128b374918138955f1c30b420938a057ec79ca589171aeb4a6b335f"} err="failed to get container status \"40bdab51d128b374918138955f1c30b420938a057ec79ca589171aeb4a6b335f\": rpc error: code = NotFound desc = could not find container \"40bdab51d128b374918138955f1c30b420938a057ec79ca589171aeb4a6b335f\": container with ID starting with 40bdab51d128b374918138955f1c30b420938a057ec79ca589171aeb4a6b335f not found: ID does not exist"
Oct 31 01:21:14.741791 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.741751    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-maas-api-tls\") pod \"0280fb12-2007-4c5c-89ca-bf2f18dc28ad\" (UID: \"0280fb12-2007-4c5c-89ca-bf2f18dc28ad\") "
Oct 31 01:21:14.741953 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.741811    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-kube-api-access-kmxzt\" (UniqueName: \"kubernetes.io/projected/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-kube-api-access-kmxzt\") pod \"0280fb12-2007-4c5c-89ca-bf2f18dc28ad\" (UID: \"0280fb12-2007-4c5c-89ca-bf2f18dc28ad\") "
Oct 31 01:21:14.741953 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.741916    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-metrics-certs\") pod \"0280fb12-2007-4c5c-89ca-bf2f18dc28ad\" (UID: \"0280fb12-2007-4c5c-89ca-bf2f18dc28ad\") "
Oct 31 01:21:14.742045 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.741990    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-openshift-service-ca\") pod \"0280fb12-2007-4c5c-89ca-bf2f18dc28ad\" (UID: \"0280fb12-2007-4c5c-89ca-bf2f18dc28ad\") "
Oct 31 01:21:14.742398 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.742369    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-openshift-service-ca" pod "0280fb12-2007-4c5c-89ca-bf2f18dc28ad" (UID: "0280fb12-2007-4c5c-89ca-bf2f18dc28ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:21:14.742549 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.742530    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54718bb1-f732-4c6b-aa42-7eda6ecec318-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:14.742614 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.742554    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgrkr\" (UniqueName: \"kubernetes.io/projected/54718bb1-f732-4c6b-aa42-7eda6ecec318-kube-api-access-lgrkr\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:14.742614 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.742569    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:14.742614 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.742586    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsmzc\" (UniqueName: \"kubernetes.io/projected/9ec0db43-a0c1-4e82-970b-dbe3f5585acd-kube-api-access-rsmzc\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:14.744013 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.743986    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-maas-api-tls" pod "0280fb12-2007-4c5c-89ca-bf2f18dc28ad" (UID: "0280fb12-2007-4c5c-89ca-bf2f18dc28ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:21:14.744100 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.744057    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-metrics-certs" pod "0280fb12-2007-4c5c-89ca-bf2f18dc28ad" (UID: "0280fb12-2007-4c5c-89ca-bf2f18dc28ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:21:14.744100 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.744091    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-kube-api-access-kmxzt" pod "0280fb12-2007-4c5c-89ca-bf2f18dc28ad" (UID: "0280fb12-2007-4c5c-89ca-bf2f18dc28ad"). InnerVolumeSpecName "kube-api-access-kmxzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:21:14.844042 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.843952    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:14.844042 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.843986    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:14.844042 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.844000    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmxzt\" (UniqueName: \"kubernetes.io/projected/0280fb12-2007-4c5c-89ca-bf2f18dc28ad-kube-api-access-kmxzt\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:14.975871 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.975829    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b"]
Oct 31 01:21:14.992161 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:14.992127    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4207cd-769675f9dc-ndr4b"]
Oct 31 01:21:15.034750 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:15.026740    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq"]
Oct 31 01:21:15.034750 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:15.034578    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4207cd-55648f6b98-x88sq"]
Oct 31 01:21:15.065405 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:15.065372    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh"]
Oct 31 01:21:15.069133 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:15.069107    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4207cd-5597f666b8-cxckh"]
Oct 31 01:21:15.293764 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:21:15.293721    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:21:15.305556 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:15.305519    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb" podUID="c975bedf-cccf-4102-bd06-d62598138f5d" containerName="payload-pre-processing" containerID="cri-o://d655d0ce9fdaefb1328609c8f15dd16f510b537562df15d3471d1a8808dcd63d" gracePeriod=25
Oct 31 01:21:15.921721 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:15.921689    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0280fb12-2007-4c5c-89ca-bf2f18dc28ad" path="/var/lib/kubelet/pods/0280fb12-2007-4c5c-89ca-bf2f18dc28ad/volumes"
Oct 31 01:21:15.922225 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:15.922208    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54718bb1-f732-4c6b-aa42-7eda6ecec318" path="/var/lib/kubelet/pods/54718bb1-f732-4c6b-aa42-7eda6ecec318/volumes"
Oct 31 01:21:15.922629 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:15.922615    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ec0db43-a0c1-4e82-970b-dbe3f5585acd" path="/var/lib/kubelet/pods/9ec0db43-a0c1-4e82-970b-dbe3f5585acd/volumes"
Oct 31 01:21:16.207508 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.207422    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"]
Oct 31 01:21:16.208104 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.208087    4327 cpu_manager.go:414] "containerMap: removing container" podUID="c6879302-3938-426c-a79b-8738dee89964" containerName="istio-proxy"
Oct 31 01:21:16.208169 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.208105    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6879302-3938-426c-a79b-8738dee89964" containerName="istio-proxy"
Oct 31 01:21:16.208169 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.208122    4327 cpu_manager.go:414] "containerMap: removing container" podUID="0280fb12-2007-4c5c-89ca-bf2f18dc28ad" containerName="maas-api"
Oct 31 01:21:16.208169 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.208127    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0280fb12-2007-4c5c-89ca-bf2f18dc28ad" containerName="maas-api"
Oct 31 01:21:16.208169 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.208141    4327 cpu_manager.go:414] "containerMap: removing container" podUID="9ec0db43-a0c1-4e82-970b-dbe3f5585acd" containerName="payload-processing"
Oct 31 01:21:16.208169 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.208147    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec0db43-a0c1-4e82-970b-dbe3f5585acd" containerName="payload-processing"
Oct 31 01:21:16.208169 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.208171    4327 cpu_manager.go:414] "containerMap: removing container" podUID="54718bb1-f732-4c6b-aa42-7eda6ecec318" containerName="payload-pre-processing"
Oct 31 01:21:16.208414 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.208177    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54718bb1-f732-4c6b-aa42-7eda6ecec318" containerName="payload-pre-processing"
Oct 31 01:21:16.208414 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.208189    4327 cpu_manager.go:414] "containerMap: removing container" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main"
Oct 31 01:21:16.208414 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.208194    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main"
Oct 31 01:21:16.208414 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.208202    4327 cpu_manager.go:414] "containerMap: removing container" podUID="b6a22f7c-91e0-44af-bcf7-f987701bc859" containerName="maas-api"
Oct 31 01:21:16.208414 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.208206    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6a22f7c-91e0-44af-bcf7-f987701bc859" containerName="maas-api"
Oct 31 01:21:16.208414 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.208304    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="0280fb12-2007-4c5c-89ca-bf2f18dc28ad" containerName="maas-api"
Oct 31 01:21:16.208414 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.208319    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec0db43-a0c1-4e82-970b-dbe3f5585acd" containerName="payload-processing"
Oct 31 01:21:16.208414 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.208331    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="54718bb1-f732-4c6b-aa42-7eda6ecec318" containerName="payload-pre-processing"
Oct 31 01:21:16.208414 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.208337    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="52f30ed7-5edf-403e-b75c-53f8b27f94bd" containerName="main"
Oct 31 01:21:16.208414 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.208342    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6a22f7c-91e0-44af-bcf7-f987701bc859" containerName="maas-api"
Oct 31 01:21:16.208414 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.208349    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6879302-3938-426c-a79b-8738dee89964" containerName="istio-proxy"
Oct 31 01:21:16.213657 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.213638    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.236044 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.236017    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-ada0c1-openshift-default-dockercfg-7hrph\""
Oct 31 01:21:16.248980 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.248950    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"]
Oct 31 01:21:16.364395 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.364349    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-envoy\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.364395 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.364399    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-credential-socket\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.364730 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.364521    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2387e9d0-6540-45b2-b624-78f6f63e26f5-istiod-ca-cert\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.364730 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.364586    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-token\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.364858 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.364737    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-podinfo\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.364858 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.364774    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nbh2\" (UniqueName: \"kubernetes.io/projected/2387e9d0-6540-45b2-b624-78f6f63e26f5-kube-api-access-4nbh2\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.364972 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.364868    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-workload-certs\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.364972 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.364944    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-data\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.365077 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.365000    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-workload-socket\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.466614 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.466523    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-data\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.466614 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.466577    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-workload-socket\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.466846 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.466709    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-envoy\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.466846 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.466739    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-credential-socket\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.466846 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.466782    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2387e9d0-6540-45b2-b624-78f6f63e26f5-istiod-ca-cert\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.466846 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.466818    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-token\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.467056 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.466911    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-podinfo\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.467056 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.466955    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nbh2\" (UniqueName: \"kubernetes.io/projected/2387e9d0-6540-45b2-b624-78f6f63e26f5-kube-api-access-4nbh2\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.467056 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.467042    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-workload-socket\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.467246 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.467096    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-credential-socket\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.467246 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.466959    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-data\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.467246 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.467226    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-workload-certs\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.467569 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.467551    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2387e9d0-6540-45b2-b624-78f6f63e26f5-istiod-ca-cert\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.467637 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.467574    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-workload-certs\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.469155 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.469127    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-envoy\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.469421 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.469403    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-podinfo\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.490863 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.490829    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-token\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.491045 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.490983    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nbh2\" (UniqueName: \"kubernetes.io/projected/2387e9d0-6540-45b2-b624-78f6f63e26f5-kube-api-access-4nbh2\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.530037 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.530003    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:16.859519 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:21:16.859489    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2387e9d0_6540_45b2_b624_78f6f63e26f5.slice/crio-919c6dd148f8ace35807f83985567167639c5054292b4c4d344c90c9aaf5bdaf WatchSource:0}: Error finding container 919c6dd148f8ace35807f83985567167639c5054292b4c4d344c90c9aaf5bdaf: Status 404 returned error can't find the container with id 919c6dd148f8ace35807f83985567167639c5054292b4c4d344c90c9aaf5bdaf
Oct 31 01:21:16.859752 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.859732    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"]
Oct 31 01:21:16.862099 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.862067    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:21:16.862185 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.862153    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:21:16.862233 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:16.862190    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:21:17.083523 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:17.083489    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb" podUID="c975bedf-cccf-4102-bd06-d62598138f5d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:21:17.664169 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:17.664134    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j" event={"ID":"2387e9d0-6540-45b2-b624-78f6f63e26f5","Type":"ContainerStarted","Data":"04cd907891d7c24397e948a3ec04a5c4bac026a2203197a7909ef12f32aad448"}
Oct 31 01:21:17.664800 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:17.664192    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j" event={"ID":"2387e9d0-6540-45b2-b624-78f6f63e26f5","Type":"ContainerStarted","Data":"919c6dd148f8ace35807f83985567167639c5054292b4c4d344c90c9aaf5bdaf"}
Oct 31 01:21:17.948597 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:17.941248    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j" podStartSLOduration=1.941227494 podStartE2EDuration="1.941227494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:17.750357611 +0000 UTC m=+3737.972393152" watchObservedRunningTime="2026-10-31 01:21:17.941227494 +0000 UTC m=+3738.163263031"
Oct 31 01:21:17.952250 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:17.951159    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9"]
Oct 31 01:21:17.952250 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:17.951532    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9" podUID="f00845e5-30d2-4bd8-89a1-67072e3fef48" containerName="maas-api" containerID="cri-o://df5c3bc45463b245ce3fb874c96acde46bcd528b1d2c34f42acc8795ab6816d8" gracePeriod=30
Oct 31 01:21:18.307706 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:18.304844    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9"
Oct 31 01:21:18.457386 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:18.453155    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f00845e5-30d2-4bd8-89a1-67072e3fef48-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f00845e5-30d2-4bd8-89a1-67072e3fef48-metrics-certs\") pod \"f00845e5-30d2-4bd8-89a1-67072e3fef48\" (UID: \"f00845e5-30d2-4bd8-89a1-67072e3fef48\") "
Oct 31 01:21:18.457386 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:18.454462    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f00845e5-30d2-4bd8-89a1-67072e3fef48-kube-api-access-qjhjg\" (UniqueName: \"kubernetes.io/projected/f00845e5-30d2-4bd8-89a1-67072e3fef48-kube-api-access-qjhjg\") pod \"f00845e5-30d2-4bd8-89a1-67072e3fef48\" (UID: \"f00845e5-30d2-4bd8-89a1-67072e3fef48\") "
Oct 31 01:21:18.457386 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:18.454530    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f00845e5-30d2-4bd8-89a1-67072e3fef48-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00845e5-30d2-4bd8-89a1-67072e3fef48-openshift-service-ca\") pod \"f00845e5-30d2-4bd8-89a1-67072e3fef48\" (UID: \"f00845e5-30d2-4bd8-89a1-67072e3fef48\") "
Oct 31 01:21:18.457386 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:18.454651    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f00845e5-30d2-4bd8-89a1-67072e3fef48-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f00845e5-30d2-4bd8-89a1-67072e3fef48-maas-api-tls\") pod \"f00845e5-30d2-4bd8-89a1-67072e3fef48\" (UID: \"f00845e5-30d2-4bd8-89a1-67072e3fef48\") "
Oct 31 01:21:18.457386 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:18.456475    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f00845e5-30d2-4bd8-89a1-67072e3fef48-openshift-service-ca" pod "f00845e5-30d2-4bd8-89a1-67072e3fef48" (UID: "f00845e5-30d2-4bd8-89a1-67072e3fef48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:21:18.458105 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:18.458017    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f00845e5-30d2-4bd8-89a1-67072e3fef48-metrics-certs" pod "f00845e5-30d2-4bd8-89a1-67072e3fef48" (UID: "f00845e5-30d2-4bd8-89a1-67072e3fef48"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:21:18.458105 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:18.458083    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f00845e5-30d2-4bd8-89a1-67072e3fef48-kube-api-access-qjhjg" pod "f00845e5-30d2-4bd8-89a1-67072e3fef48" (UID: "f00845e5-30d2-4bd8-89a1-67072e3fef48"). InnerVolumeSpecName "kube-api-access-qjhjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:21:18.458462 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:18.458428    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f00845e5-30d2-4bd8-89a1-67072e3fef48-maas-api-tls" pod "f00845e5-30d2-4bd8-89a1-67072e3fef48" (UID: "f00845e5-30d2-4bd8-89a1-67072e3fef48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:21:18.535764 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:18.531427    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:18.535764 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:18.533769    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Oct 31 01:21:18.535764 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:18.533827    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j" podUID="2387e9d0-6540-45b2-b624-78f6f63e26f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Oct 31 01:21:18.556483 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:18.556443    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjhjg\" (UniqueName: \"kubernetes.io/projected/f00845e5-30d2-4bd8-89a1-67072e3fef48-kube-api-access-qjhjg\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:18.556483 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:18.556484    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00845e5-30d2-4bd8-89a1-67072e3fef48-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:18.556737 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:18.556500    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f00845e5-30d2-4bd8-89a1-67072e3fef48-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:18.556737 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:18.556514    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f00845e5-30d2-4bd8-89a1-67072e3fef48-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:18.671908 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:18.671869    4327 generic.go:356] "Generic (PLEG): container finished" podID="f00845e5-30d2-4bd8-89a1-67072e3fef48" containerID="df5c3bc45463b245ce3fb874c96acde46bcd528b1d2c34f42acc8795ab6816d8" exitCode=0
Oct 31 01:21:18.672331 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:18.671919    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9"
Oct 31 01:21:18.672331 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:18.671918    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9" event={"ID":"f00845e5-30d2-4bd8-89a1-67072e3fef48","Type":"ContainerDied","Data":"df5c3bc45463b245ce3fb874c96acde46bcd528b1d2c34f42acc8795ab6816d8"}
Oct 31 01:21:18.672331 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:18.671982    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9" event={"ID":"f00845e5-30d2-4bd8-89a1-67072e3fef48","Type":"ContainerDied","Data":"605b8a6b10916d9c6f568dd53c821c26bb9acbe3cdc4eab7501d7a481ad62768"}
Oct 31 01:21:18.672331 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:18.672006    4327 scope.go:122] "RemoveContainer" containerID="df5c3bc45463b245ce3fb874c96acde46bcd528b1d2c34f42acc8795ab6816d8"
Oct 31 01:21:18.682658 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:18.682639    4327 scope.go:122] "RemoveContainer" containerID="df5c3bc45463b245ce3fb874c96acde46bcd528b1d2c34f42acc8795ab6816d8"
Oct 31 01:21:18.682955 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:21:18.682935    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df5c3bc45463b245ce3fb874c96acde46bcd528b1d2c34f42acc8795ab6816d8\": container with ID starting with df5c3bc45463b245ce3fb874c96acde46bcd528b1d2c34f42acc8795ab6816d8 not found: ID does not exist" containerID="df5c3bc45463b245ce3fb874c96acde46bcd528b1d2c34f42acc8795ab6816d8"
Oct 31 01:21:18.683010 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:18.682964    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df5c3bc45463b245ce3fb874c96acde46bcd528b1d2c34f42acc8795ab6816d8"} err="failed to get container status \"df5c3bc45463b245ce3fb874c96acde46bcd528b1d2c34f42acc8795ab6816d8\": rpc error: code = NotFound desc = could not find container \"df5c3bc45463b245ce3fb874c96acde46bcd528b1d2c34f42acc8795ab6816d8\": container with ID starting with df5c3bc45463b245ce3fb874c96acde46bcd528b1d2c34f42acc8795ab6816d8 not found: ID does not exist"
Oct 31 01:21:18.746960 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:18.746919    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9"]
Oct 31 01:21:18.751322 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:18.751291    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b024b5-95ccd9b45-6dvx9"]
Oct 31 01:21:19.538706 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:19.538163    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Oct 31 01:21:19.538706 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:19.538245    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j" podUID="2387e9d0-6540-45b2-b624-78f6f63e26f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Oct 31 01:21:19.947869 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:19.947822    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f00845e5-30d2-4bd8-89a1-67072e3fef48" path="/var/lib/kubelet/pods/f00845e5-30d2-4bd8-89a1-67072e3fef48/volumes"
Oct 31 01:21:20.537705 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:20.536140    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Oct 31 01:21:20.537705 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:20.536226    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j" podUID="2387e9d0-6540-45b2-b624-78f6f63e26f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Oct 31 01:21:21.542131 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:21.542077    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Oct 31 01:21:21.542666 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:21.542161    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j" podUID="2387e9d0-6540-45b2-b624-78f6f63e26f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Oct 31 01:21:22.538705 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:22.538143    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Oct 31 01:21:22.538705 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:22.538220    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j" podUID="2387e9d0-6540-45b2-b624-78f6f63e26f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Oct 31 01:21:23.667533 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:23.667496    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:23.669405 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:23.669305    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:23.672121 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:23.672094    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:21:23.853032 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:23.852997    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf"]
Oct 31 01:21:23.853225 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:23.853204    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf" podUID="b7f71dd1-1b1e-4c68-8a2d-8641ea516faf" containerName="maas-api" containerID="cri-o://e384579678c4de3cba6d8d7156a19866de0baa603a8e37ce0da881755c7ff7e7" gracePeriod=30
Oct 31 01:21:24.192325 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:24.192297    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf"
Oct 31 01:21:24.239406 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:24.238199    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-metrics-certs\") pod \"b7f71dd1-1b1e-4c68-8a2d-8641ea516faf\" (UID: \"b7f71dd1-1b1e-4c68-8a2d-8641ea516faf\") "
Oct 31 01:21:24.239406 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:24.238335    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-kube-api-access-ggpmd\" (UniqueName: \"kubernetes.io/projected/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-kube-api-access-ggpmd\") pod \"b7f71dd1-1b1e-4c68-8a2d-8641ea516faf\" (UID: \"b7f71dd1-1b1e-4c68-8a2d-8641ea516faf\") "
Oct 31 01:21:24.239406 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:24.238390    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-maas-api-tls\") pod \"b7f71dd1-1b1e-4c68-8a2d-8641ea516faf\" (UID: \"b7f71dd1-1b1e-4c68-8a2d-8641ea516faf\") "
Oct 31 01:21:24.239406 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:24.238530    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-openshift-service-ca\") pod \"b7f71dd1-1b1e-4c68-8a2d-8641ea516faf\" (UID: \"b7f71dd1-1b1e-4c68-8a2d-8641ea516faf\") "
Oct 31 01:21:24.240273 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:24.239952    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-openshift-service-ca" pod "b7f71dd1-1b1e-4c68-8a2d-8641ea516faf" (UID: "b7f71dd1-1b1e-4c68-8a2d-8641ea516faf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:21:24.261695 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:24.261645    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-kube-api-access-ggpmd" pod "b7f71dd1-1b1e-4c68-8a2d-8641ea516faf" (UID: "b7f71dd1-1b1e-4c68-8a2d-8641ea516faf"). InnerVolumeSpecName "kube-api-access-ggpmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:21:24.269374 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:24.269245    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-maas-api-tls" pod "b7f71dd1-1b1e-4c68-8a2d-8641ea516faf" (UID: "b7f71dd1-1b1e-4c68-8a2d-8641ea516faf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:21:24.269374 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:24.269328    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-metrics-certs" pod "b7f71dd1-1b1e-4c68-8a2d-8641ea516faf" (UID: "b7f71dd1-1b1e-4c68-8a2d-8641ea516faf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:21:24.340695 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:24.340640    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:24.340695 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:24.340689    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggpmd\" (UniqueName: \"kubernetes.io/projected/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-kube-api-access-ggpmd\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:24.340695 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:24.340700    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:24.340946 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:24.340709    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:24.722628 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:24.722589    4327 generic.go:356] "Generic (PLEG): container finished" podID="b7f71dd1-1b1e-4c68-8a2d-8641ea516faf" containerID="e384579678c4de3cba6d8d7156a19866de0baa603a8e37ce0da881755c7ff7e7" exitCode=0
Oct 31 01:21:24.723140 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:24.722661    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf"
Oct 31 01:21:24.723140 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:24.722713    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf" event={"ID":"b7f71dd1-1b1e-4c68-8a2d-8641ea516faf","Type":"ContainerDied","Data":"e384579678c4de3cba6d8d7156a19866de0baa603a8e37ce0da881755c7ff7e7"}
Oct 31 01:21:24.723140 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:24.722774    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf" event={"ID":"b7f71dd1-1b1e-4c68-8a2d-8641ea516faf","Type":"ContainerDied","Data":"9f35269253a82fed8123a82bf5d4ba0565be019264a5e7efbe2d534d427df297"}
Oct 31 01:21:24.723140 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:24.722805    4327 scope.go:122] "RemoveContainer" containerID="e384579678c4de3cba6d8d7156a19866de0baa603a8e37ce0da881755c7ff7e7"
Oct 31 01:21:24.734440 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:24.734415    4327 scope.go:122] "RemoveContainer" containerID="e384579678c4de3cba6d8d7156a19866de0baa603a8e37ce0da881755c7ff7e7"
Oct 31 01:21:24.734823 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:21:24.734799    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e384579678c4de3cba6d8d7156a19866de0baa603a8e37ce0da881755c7ff7e7\": container with ID starting with e384579678c4de3cba6d8d7156a19866de0baa603a8e37ce0da881755c7ff7e7 not found: ID does not exist" containerID="e384579678c4de3cba6d8d7156a19866de0baa603a8e37ce0da881755c7ff7e7"
Oct 31 01:21:24.734906 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:24.734830    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e384579678c4de3cba6d8d7156a19866de0baa603a8e37ce0da881755c7ff7e7"} err="failed to get container status \"e384579678c4de3cba6d8d7156a19866de0baa603a8e37ce0da881755c7ff7e7\": rpc error: code = NotFound desc = could not find container \"e384579678c4de3cba6d8d7156a19866de0baa603a8e37ce0da881755c7ff7e7\": container with ID starting with e384579678c4de3cba6d8d7156a19866de0baa603a8e37ce0da881755c7ff7e7 not found: ID does not exist"
Oct 31 01:21:24.783566 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:24.783531    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf"]
Oct 31 01:21:24.792263 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:24.792229    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-bcdb4e-fdff6db48-wfrnf"]
Oct 31 01:21:25.425746 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.425707    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"]
Oct 31 01:21:25.426428 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.426406    4327 cpu_manager.go:414] "containerMap: removing container" podUID="f00845e5-30d2-4bd8-89a1-67072e3fef48" containerName="maas-api"
Oct 31 01:21:25.426428 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.426428    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f00845e5-30d2-4bd8-89a1-67072e3fef48" containerName="maas-api"
Oct 31 01:21:25.426615 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.426443    4327 cpu_manager.go:414] "containerMap: removing container" podUID="b7f71dd1-1b1e-4c68-8a2d-8641ea516faf" containerName="maas-api"
Oct 31 01:21:25.426615 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.426451    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f71dd1-1b1e-4c68-8a2d-8641ea516faf" containerName="maas-api"
Oct 31 01:21:25.426615 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.426595    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="f00845e5-30d2-4bd8-89a1-67072e3fef48" containerName="maas-api"
Oct 31 01:21:25.426615 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.426609    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7f71dd1-1b1e-4c68-8a2d-8641ea516faf" containerName="maas-api"
Oct 31 01:21:25.431623 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.431600    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.456061 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.456028    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"]
Oct 31 01:21:25.463913 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.463879    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-d10721-openshift-default-dockercfg-f7nsh\""
Oct 31 01:21:25.554737 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.554697    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-istio-data\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.554951 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.554833    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsrpz\" (UniqueName: \"kubernetes.io/projected/f745fa94-dca6-46ac-981e-73ce0831e867-kube-api-access-lsrpz\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.554951 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.554879    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-workload-certs\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.555071 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.555044    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f745fa94-dca6-46ac-981e-73ce0831e867-istio-podinfo\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.555120 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.555084    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-credential-socket\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.555184 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.555127    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f745fa94-dca6-46ac-981e-73ce0831e867-istiod-ca-cert\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.555251 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.555239    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-workload-socket\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.555294 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.555265    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-istio-envoy\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.555347 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.555292    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f745fa94-dca6-46ac-981e-73ce0831e867-istio-token\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.656459 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.656415    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsrpz\" (UniqueName: \"kubernetes.io/projected/f745fa94-dca6-46ac-981e-73ce0831e867-kube-api-access-lsrpz\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.656459 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.656460    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-workload-certs\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.656710 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.656542    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f745fa94-dca6-46ac-981e-73ce0831e867-istio-podinfo\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.656710 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.656578    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-credential-socket\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.656710 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.656607    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f745fa94-dca6-46ac-981e-73ce0831e867-istiod-ca-cert\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.656710 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.656658    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-workload-socket\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.656710 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.656703    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-istio-envoy\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.656974 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.656743    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f745fa94-dca6-46ac-981e-73ce0831e867-istio-token\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.656974 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.656822    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-istio-data\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.657236 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.657116    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-workload-socket\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.657236 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.657185    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-workload-certs\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.657236 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.657199    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-istio-data\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.657496 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.657475    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-credential-socket\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.657570 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.657551    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f745fa94-dca6-46ac-981e-73ce0831e867-istiod-ca-cert\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.659098 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.659079    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-istio-envoy\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.659343 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.659327    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f745fa94-dca6-46ac-981e-73ce0831e867-istio-podinfo\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.682628 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.682559    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f745fa94-dca6-46ac-981e-73ce0831e867-istio-token\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.682758 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.682703    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsrpz\" (UniqueName: \"kubernetes.io/projected/f745fa94-dca6-46ac-981e-73ce0831e867-kube-api-access-lsrpz\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.745640 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.745609    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:25.922589 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:25.922555    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7f71dd1-1b1e-4c68-8a2d-8641ea516faf" path="/var/lib/kubelet/pods/b7f71dd1-1b1e-4c68-8a2d-8641ea516faf/volumes"
Oct 31 01:21:26.063247 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:26.063218    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"]
Oct 31 01:21:26.063920 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:21:26.063897    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf745fa94_dca6_46ac_981e_73ce0831e867.slice/crio-f31068e2d23c0c0cb31f41087c319f4eb969c0069bc67420f7024e83f280d542 WatchSource:0}: Error finding container f31068e2d23c0c0cb31f41087c319f4eb969c0069bc67420f7024e83f280d542: Status 404 returned error can't find the container with id f31068e2d23c0c0cb31f41087c319f4eb969c0069bc67420f7024e83f280d542
Oct 31 01:21:26.067158 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:26.067117    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:21:26.067232 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:26.067200    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:21:26.067283 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:26.067242    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:21:26.748567 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:26.748527    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7" event={"ID":"f745fa94-dca6-46ac-981e-73ce0831e867","Type":"ContainerStarted","Data":"8bb830716b53d48692fb82190c7393afdd08a6fb0f43696b85306ffc25db8f54"}
Oct 31 01:21:26.749086 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:26.748590    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7" event={"ID":"f745fa94-dca6-46ac-981e-73ce0831e867","Type":"ContainerStarted","Data":"f31068e2d23c0c0cb31f41087c319f4eb969c0069bc67420f7024e83f280d542"}
Oct 31 01:21:26.795121 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:26.795068    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7" podStartSLOduration=1.795052637 podStartE2EDuration="1.795052637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:26.793737019 +0000 UTC m=+3747.015772560" watchObservedRunningTime="2026-10-31 01:21:26.795052637 +0000 UTC m=+3747.017088179"
Oct 31 01:21:27.083806 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:27.083716    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb" podUID="c975bedf-cccf-4102-bd06-d62598138f5d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:21:27.254244 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:27.254205    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"]
Oct 31 01:21:27.254471 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:27.254442    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq" podUID="a24c7c0a-782b-4d04-976d-9f62a964250d" containerName="istio-proxy" containerID="cri-o://903646467c97419f3a1be016e12ed875f2c66114cde668fd93e6ce2eb34b22d3" gracePeriod=30
Oct 31 01:21:27.746053 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:27.746018    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:27.747978 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:27.747950    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Oct 31 01:21:27.748135 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:27.748003    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7" podUID="f745fa94-dca6-46ac-981e-73ce0831e867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Oct 31 01:21:28.747014 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:28.746980    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Oct 31 01:21:28.747557 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:28.747033    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7" podUID="f745fa94-dca6-46ac-981e-73ce0831e867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Oct 31 01:21:29.746474 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:29.746434    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Oct 31 01:21:29.746632 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:29.746504    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7" podUID="f745fa94-dca6-46ac-981e-73ce0831e867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Oct 31 01:21:30.746220 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:30.746181    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Oct 31 01:21:30.746644 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:30.746256    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7" podUID="f745fa94-dca6-46ac-981e-73ce0831e867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Oct 31 01:21:30.913076 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:30.913036    4327 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:21:30.913261 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:30.913092    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq" podUID="a24c7c0a-782b-4d04-976d-9f62a964250d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:21:31.746459 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:31.746411    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Oct 31 01:21:31.746905 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:31.746495    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7" podUID="f745fa94-dca6-46ac-981e-73ce0831e867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Oct 31 01:21:32.558570 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.558527    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl"]
Oct 31 01:21:32.563907 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.563882    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl"
Oct 31 01:21:32.585768 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.585730    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl"]
Oct 31 01:21:32.593381 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.593291    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-8f0029-serving-cert\""
Oct 31 01:21:32.596628 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.596122    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 01:21:32.698508 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.698473    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a24c7c0a-782b-4d04-976d-9f62a964250d-kube-api-access-g9wkj\" (UniqueName: \"kubernetes.io/projected/a24c7c0a-782b-4d04-976d-9f62a964250d-kube-api-access-g9wkj\") pod \"a24c7c0a-782b-4d04-976d-9f62a964250d\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") "
Oct 31 01:21:32.698754 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.698553    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-data\") pod \"a24c7c0a-782b-4d04-976d-9f62a964250d\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") "
Oct 31 01:21:32.698754 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.698586    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-workload-certs\") pod \"a24c7c0a-782b-4d04-976d-9f62a964250d\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") "
Oct 31 01:21:32.698754 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.698655    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-workload-socket\") pod \"a24c7c0a-782b-4d04-976d-9f62a964250d\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") "
Oct 31 01:21:32.698893 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.698774    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-podinfo\") pod \"a24c7c0a-782b-4d04-976d-9f62a964250d\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") "
Oct 31 01:21:32.698893 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.698864    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-envoy\") pod \"a24c7c0a-782b-4d04-976d-9f62a964250d\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") "
Oct 31 01:21:32.698962 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.698896    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-data" pod "a24c7c0a-782b-4d04-976d-9f62a964250d" (UID: "a24c7c0a-782b-4d04-976d-9f62a964250d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:21:32.699032 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.699010    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a24c7c0a-782b-4d04-976d-9f62a964250d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a24c7c0a-782b-4d04-976d-9f62a964250d-istiod-ca-cert\") pod \"a24c7c0a-782b-4d04-976d-9f62a964250d\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") "
Oct 31 01:21:32.699120 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.699097    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-workload-socket" pod "a24c7c0a-782b-4d04-976d-9f62a964250d" (UID: "a24c7c0a-782b-4d04-976d-9f62a964250d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:21:32.699219 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.699199    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-credential-socket\") pod \"a24c7c0a-782b-4d04-976d-9f62a964250d\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") "
Oct 31 01:21:32.699292 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.699279    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-token\" (UniqueName: \"kubernetes.io/projected/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-token\") pod \"a24c7c0a-782b-4d04-976d-9f62a964250d\" (UID: \"a24c7c0a-782b-4d04-976d-9f62a964250d\") "
Oct 31 01:21:32.699330 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.699309    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-workload-certs" pod "a24c7c0a-782b-4d04-976d-9f62a964250d" (UID: "a24c7c0a-782b-4d04-976d-9f62a964250d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:21:32.699581 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.699552    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tftcx\" (UniqueName: \"kubernetes.io/projected/a33d24c7-5486-4dba-aca9-83bedd6b6464-kube-api-access-tftcx\") pod \"maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl\" (UID: \"a33d24c7-5486-4dba-aca9-83bedd6b6464\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl"
Oct 31 01:21:32.702568 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.702524    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-credential-socket" pod "a24c7c0a-782b-4d04-976d-9f62a964250d" (UID: "a24c7c0a-782b-4d04-976d-9f62a964250d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:21:32.703185 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.703133    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a24c7c0a-782b-4d04-976d-9f62a964250d-istiod-ca-cert" pod "a24c7c0a-782b-4d04-976d-9f62a964250d" (UID: "a24c7c0a-782b-4d04-976d-9f62a964250d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:21:32.703574 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.703537    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a33d24c7-5486-4dba-aca9-83bedd6b6464-metrics-certs\") pod \"maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl\" (UID: \"a33d24c7-5486-4dba-aca9-83bedd6b6464\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl"
Oct 31 01:21:32.703846 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.703813    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a33d24c7-5486-4dba-aca9-83bedd6b6464-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl\" (UID: \"a33d24c7-5486-4dba-aca9-83bedd6b6464\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl"
Oct 31 01:21:32.704023 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.703997    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a33d24c7-5486-4dba-aca9-83bedd6b6464-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl\" (UID: \"a33d24c7-5486-4dba-aca9-83bedd6b6464\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl"
Oct 31 01:21:32.704237 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.704217    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:32.704368 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.704245    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:32.704368 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.704261    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:32.704368 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.704275    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a24c7c0a-782b-4d04-976d-9f62a964250d-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:32.704368 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.704296    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:32.705799 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.704734    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-token" pod "a24c7c0a-782b-4d04-976d-9f62a964250d" (UID: "a24c7c0a-782b-4d04-976d-9f62a964250d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:21:32.772718 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.771958    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:32.773315 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.773247    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:32.774452 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.774423    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:21:32.774739 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.774647    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a24c7c0a-782b-4d04-976d-9f62a964250d-kube-api-access-g9wkj" pod "a24c7c0a-782b-4d04-976d-9f62a964250d" (UID: "a24c7c0a-782b-4d04-976d-9f62a964250d"). InnerVolumeSpecName "kube-api-access-g9wkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:21:32.775118 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.775090    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-envoy" pod "a24c7c0a-782b-4d04-976d-9f62a964250d" (UID: "a24c7c0a-782b-4d04-976d-9f62a964250d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:21:32.777059 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.777031    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-podinfo" pod "a24c7c0a-782b-4d04-976d-9f62a964250d" (UID: "a24c7c0a-782b-4d04-976d-9f62a964250d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:21:32.790251 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.790180    4327 generic.go:356] "Generic (PLEG): container finished" podID="a24c7c0a-782b-4d04-976d-9f62a964250d" containerID="903646467c97419f3a1be016e12ed875f2c66114cde668fd93e6ce2eb34b22d3" exitCode=0
Oct 31 01:21:32.790396 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.790252    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"
Oct 31 01:21:32.790396 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.790260    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq" event={"ID":"a24c7c0a-782b-4d04-976d-9f62a964250d","Type":"ContainerDied","Data":"903646467c97419f3a1be016e12ed875f2c66114cde668fd93e6ce2eb34b22d3"}
Oct 31 01:21:32.790396 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.790314    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq" event={"ID":"a24c7c0a-782b-4d04-976d-9f62a964250d","Type":"ContainerDied","Data":"2676cb3206573b1c673f693f9f61badbd6707c7b7ee611a247bb98fc0f3d9469"}
Oct 31 01:21:32.790396 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.790338    4327 scope.go:122] "RemoveContainer" containerID="903646467c97419f3a1be016e12ed875f2c66114cde668fd93e6ce2eb34b22d3"
Oct 31 01:21:32.802416 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.802391    4327 scope.go:122] "RemoveContainer" containerID="eec57e7ca9466cfe7d8f9347b97438dcd961993c5c26bdc11a5ff5ff7a77a272"
Oct 31 01:21:32.804957 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.804938    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a33d24c7-5486-4dba-aca9-83bedd6b6464-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl\" (UID: \"a33d24c7-5486-4dba-aca9-83bedd6b6464\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl"
Oct 31 01:21:32.805037 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.805021    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tftcx\" (UniqueName: \"kubernetes.io/projected/a33d24c7-5486-4dba-aca9-83bedd6b6464-kube-api-access-tftcx\") pod \"maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl\" (UID: \"a33d24c7-5486-4dba-aca9-83bedd6b6464\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl"
Oct 31 01:21:32.805078 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.805067    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a33d24c7-5486-4dba-aca9-83bedd6b6464-metrics-certs\") pod \"maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl\" (UID: \"a33d24c7-5486-4dba-aca9-83bedd6b6464\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl"
Oct 31 01:21:32.805141 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.805130    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a33d24c7-5486-4dba-aca9-83bedd6b6464-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl\" (UID: \"a33d24c7-5486-4dba-aca9-83bedd6b6464\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl"
Oct 31 01:21:32.805211 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.805201    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:32.805271 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.805213    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:32.805271 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.805221    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9wkj\" (UniqueName: \"kubernetes.io/projected/a24c7c0a-782b-4d04-976d-9f62a964250d-kube-api-access-g9wkj\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:32.805271 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.805230    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a24c7c0a-782b-4d04-976d-9f62a964250d-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:32.805428 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:21:32.805319    4327 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-serving-cert: secret "maas-api-e2e-ipp-b-8f0029-serving-cert" not found
Oct 31 01:21:32.805428 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:21:32.805406    4327 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a33d24c7-5486-4dba-aca9-83bedd6b6464-maas-api-tls podName:a33d24c7-5486-4dba-aca9-83bedd6b6464 nodeName:}" failed. No retries permitted until 2026-10-31 01:21:33.305388489 +0000 UTC m=+3753.527424017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a33d24c7-5486-4dba-aca9-83bedd6b6464-maas-api-tls") pod "maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl" (UID: "a33d24c7-5486-4dba-aca9-83bedd6b6464") : secret "maas-api-e2e-ipp-b-8f0029-serving-cert" not found
Oct 31 01:21:32.805751 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.805722    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a33d24c7-5486-4dba-aca9-83bedd6b6464-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl\" (UID: \"a33d24c7-5486-4dba-aca9-83bedd6b6464\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl"
Oct 31 01:21:32.807616 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.807596    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a33d24c7-5486-4dba-aca9-83bedd6b6464-metrics-certs\") pod \"maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl\" (UID: \"a33d24c7-5486-4dba-aca9-83bedd6b6464\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl"
Oct 31 01:21:32.812426 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.812404    4327 scope.go:122] "RemoveContainer" containerID="903646467c97419f3a1be016e12ed875f2c66114cde668fd93e6ce2eb34b22d3"
Oct 31 01:21:32.812751 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:21:32.812724    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"903646467c97419f3a1be016e12ed875f2c66114cde668fd93e6ce2eb34b22d3\": container with ID starting with 903646467c97419f3a1be016e12ed875f2c66114cde668fd93e6ce2eb34b22d3 not found: ID does not exist" containerID="903646467c97419f3a1be016e12ed875f2c66114cde668fd93e6ce2eb34b22d3"
Oct 31 01:21:32.812855 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.812761    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"903646467c97419f3a1be016e12ed875f2c66114cde668fd93e6ce2eb34b22d3"} err="failed to get container status \"903646467c97419f3a1be016e12ed875f2c66114cde668fd93e6ce2eb34b22d3\": rpc error: code = NotFound desc = could not find container \"903646467c97419f3a1be016e12ed875f2c66114cde668fd93e6ce2eb34b22d3\": container with ID starting with 903646467c97419f3a1be016e12ed875f2c66114cde668fd93e6ce2eb34b22d3 not found: ID does not exist"
Oct 31 01:21:32.812855 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.812789    4327 scope.go:122] "RemoveContainer" containerID="eec57e7ca9466cfe7d8f9347b97438dcd961993c5c26bdc11a5ff5ff7a77a272"
Oct 31 01:21:32.813129 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:21:32.813094    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eec57e7ca9466cfe7d8f9347b97438dcd961993c5c26bdc11a5ff5ff7a77a272\": container with ID starting with eec57e7ca9466cfe7d8f9347b97438dcd961993c5c26bdc11a5ff5ff7a77a272 not found: ID does not exist" containerID="eec57e7ca9466cfe7d8f9347b97438dcd961993c5c26bdc11a5ff5ff7a77a272"
Oct 31 01:21:32.813191 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.813127    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eec57e7ca9466cfe7d8f9347b97438dcd961993c5c26bdc11a5ff5ff7a77a272"} err="failed to get container status \"eec57e7ca9466cfe7d8f9347b97438dcd961993c5c26bdc11a5ff5ff7a77a272\": rpc error: code = NotFound desc = could not find container \"eec57e7ca9466cfe7d8f9347b97438dcd961993c5c26bdc11a5ff5ff7a77a272\": container with ID starting with eec57e7ca9466cfe7d8f9347b97438dcd961993c5c26bdc11a5ff5ff7a77a272 not found: ID does not exist"
Oct 31 01:21:32.832032 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.831999    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tftcx\" (UniqueName: \"kubernetes.io/projected/a33d24c7-5486-4dba-aca9-83bedd6b6464-kube-api-access-tftcx\") pod \"maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl\" (UID: \"a33d24c7-5486-4dba-aca9-83bedd6b6464\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl"
Oct 31 01:21:32.924519 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.924484    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"]
Oct 31 01:21:32.930356 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:32.930321    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-xtjwq"]
Oct 31 01:21:33.313357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:33.313323    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a33d24c7-5486-4dba-aca9-83bedd6b6464-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl\" (UID: \"a33d24c7-5486-4dba-aca9-83bedd6b6464\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl"
Oct 31 01:21:33.316336 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:33.316302    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a33d24c7-5486-4dba-aca9-83bedd6b6464-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl\" (UID: \"a33d24c7-5486-4dba-aca9-83bedd6b6464\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl"
Oct 31 01:21:33.500723 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:33.500672    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl"
Oct 31 01:21:33.819039 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:33.818932    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl"]
Oct 31 01:21:33.823846 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:21:33.823800    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33d24c7_5486_4dba_aca9_83bedd6b6464.slice/crio-82fee2f7326a8723f5a29d2ceaabd38e7a66476a7566987c484102cc41de3767 WatchSource:0}: Error finding container 82fee2f7326a8723f5a29d2ceaabd38e7a66476a7566987c484102cc41de3767: Status 404 returned error can't find the container with id 82fee2f7326a8723f5a29d2ceaabd38e7a66476a7566987c484102cc41de3767
Oct 31 01:21:33.923628 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:33.923586    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a24c7c0a-782b-4d04-976d-9f62a964250d" path="/var/lib/kubelet/pods/a24c7c0a-782b-4d04-976d-9f62a964250d/volumes"
Oct 31 01:21:34.826298 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:34.826263    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl" event={"ID":"a33d24c7-5486-4dba-aca9-83bedd6b6464","Type":"ContainerStarted","Data":"45640486e52c2c898971be8df68d850c7c3d4f94300da678ef7fd2f5e7c2865a"}
Oct 31 01:21:34.826719 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:34.826317    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl" event={"ID":"a33d24c7-5486-4dba-aca9-83bedd6b6464","Type":"ContainerStarted","Data":"82fee2f7326a8723f5a29d2ceaabd38e7a66476a7566987c484102cc41de3767"}
Oct 31 01:21:34.826719 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:34.826414    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl"
Oct 31 01:21:34.872568 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:34.872500    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl" podStartSLOduration=2.494817694 podStartE2EDuration="2.872485521s" totalImagesPullingTime="377.667827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:34.869457801 +0000 UTC m=+3755.091493355" watchObservedRunningTime="2026-10-31 01:21:34.872485521 +0000 UTC m=+3755.094521090"
Oct 31 01:21:35.572511 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:35.572457    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb"
Oct 31 01:21:35.638933 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:35.638901    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c975bedf-cccf-4102-bd06-d62598138f5d-kube-api-access-r9n55\" (UniqueName: \"kubernetes.io/projected/c975bedf-cccf-4102-bd06-d62598138f5d-kube-api-access-r9n55\") pod \"c975bedf-cccf-4102-bd06-d62598138f5d\" (UID: \"c975bedf-cccf-4102-bd06-d62598138f5d\") "
Oct 31 01:21:35.639142 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:35.639120    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c975bedf-cccf-4102-bd06-d62598138f5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c975bedf-cccf-4102-bd06-d62598138f5d-plugins-config-volume\") pod \"c975bedf-cccf-4102-bd06-d62598138f5d\" (UID: \"c975bedf-cccf-4102-bd06-d62598138f5d\") "
Oct 31 01:21:35.639495 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:35.639471    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c975bedf-cccf-4102-bd06-d62598138f5d-plugins-config-volume" pod "c975bedf-cccf-4102-bd06-d62598138f5d" (UID: "c975bedf-cccf-4102-bd06-d62598138f5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:21:35.641451 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:35.641422    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c975bedf-cccf-4102-bd06-d62598138f5d-kube-api-access-r9n55" pod "c975bedf-cccf-4102-bd06-d62598138f5d" (UID: "c975bedf-cccf-4102-bd06-d62598138f5d"). InnerVolumeSpecName "kube-api-access-r9n55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:21:35.740803 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:35.740768    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9n55\" (UniqueName: \"kubernetes.io/projected/c975bedf-cccf-4102-bd06-d62598138f5d-kube-api-access-r9n55\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:35.740803 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:35.740796    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c975bedf-cccf-4102-bd06-d62598138f5d-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:21:35.834568 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:35.834480    4327 generic.go:356] "Generic (PLEG): container finished" podID="c975bedf-cccf-4102-bd06-d62598138f5d" containerID="d655d0ce9fdaefb1328609c8f15dd16f510b537562df15d3471d1a8808dcd63d" exitCode=0
Oct 31 01:21:35.834568 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:35.834516    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb" event={"ID":"c975bedf-cccf-4102-bd06-d62598138f5d","Type":"ContainerDied","Data":"d655d0ce9fdaefb1328609c8f15dd16f510b537562df15d3471d1a8808dcd63d"}
Oct 31 01:21:35.834568 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:35.834531    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb"
Oct 31 01:21:35.835126 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:35.834576    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb" event={"ID":"c975bedf-cccf-4102-bd06-d62598138f5d","Type":"ContainerDied","Data":"69de28b595792d6466496f6a3605f29ae7737d4e0ba36fb31bda9e227e86d977"}
Oct 31 01:21:35.835126 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:35.834600    4327 scope.go:122] "RemoveContainer" containerID="d655d0ce9fdaefb1328609c8f15dd16f510b537562df15d3471d1a8808dcd63d"
Oct 31 01:21:35.845297 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:35.845273    4327 scope.go:122] "RemoveContainer" containerID="d655d0ce9fdaefb1328609c8f15dd16f510b537562df15d3471d1a8808dcd63d"
Oct 31 01:21:35.845654 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:21:35.845633    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d655d0ce9fdaefb1328609c8f15dd16f510b537562df15d3471d1a8808dcd63d\": container with ID starting with d655d0ce9fdaefb1328609c8f15dd16f510b537562df15d3471d1a8808dcd63d not found: ID does not exist" containerID="d655d0ce9fdaefb1328609c8f15dd16f510b537562df15d3471d1a8808dcd63d"
Oct 31 01:21:35.845727 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:35.845663    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d655d0ce9fdaefb1328609c8f15dd16f510b537562df15d3471d1a8808dcd63d"} err="failed to get container status \"d655d0ce9fdaefb1328609c8f15dd16f510b537562df15d3471d1a8808dcd63d\": rpc error: code = NotFound desc = could not find container \"d655d0ce9fdaefb1328609c8f15dd16f510b537562df15d3471d1a8808dcd63d\": container with ID starting with d655d0ce9fdaefb1328609c8f15dd16f510b537562df15d3471d1a8808dcd63d not found: ID does not exist"
Oct 31 01:21:35.887597 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:35.887565    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb"]
Oct 31 01:21:35.899984 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:35.899950    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-bcdb4e-5497455cf-rnptb"]
Oct 31 01:21:35.921674 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:35.921645    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c975bedf-cccf-4102-bd06-d62598138f5d" path="/var/lib/kubelet/pods/c975bedf-cccf-4102-bd06-d62598138f5d/volumes"
Oct 31 01:21:40.841018 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:40.840985    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl"
Oct 31 01:21:43.375557 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:43.375519    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx"]
Oct 31 01:21:45.294019 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:21:45.293978    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:21:48.376577 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:48.376509    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx" podUID="eab68ddb-030b-40ba-a1f9-bac24591237e" containerName="payload-pre-processing" containerID="cri-o://ab8a8b3f59b553f92097ccd7ebc677c631e39eefa02ce1c0e536f713408ed119" gracePeriod=25
Oct 31 01:21:51.711548 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:51.711504    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"]
Oct 31 01:21:55.374530 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:21:55.374491    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx" podUID="eab68ddb-030b-40ba-a1f9-bac24591237e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:22:02.342081 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.342026    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"]
Oct 31 01:22:02.342926 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.342901    4327 cpu_manager.go:414] "containerMap: removing container" podUID="c975bedf-cccf-4102-bd06-d62598138f5d" containerName="payload-pre-processing"
Oct 31 01:22:02.342926 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.342926    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c975bedf-cccf-4102-bd06-d62598138f5d" containerName="payload-pre-processing"
Oct 31 01:22:02.343116 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.342957    4327 cpu_manager.go:414] "containerMap: removing container" podUID="a24c7c0a-782b-4d04-976d-9f62a964250d" containerName="istio-proxy"
Oct 31 01:22:02.343116 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.342965    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a24c7c0a-782b-4d04-976d-9f62a964250d" containerName="istio-proxy"
Oct 31 01:22:02.343116 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.343002    4327 cpu_manager.go:414] "containerMap: removing container" podUID="a24c7c0a-782b-4d04-976d-9f62a964250d" containerName="istio-proxy"
Oct 31 01:22:02.343116 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.343010    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a24c7c0a-782b-4d04-976d-9f62a964250d" containerName="istio-proxy"
Oct 31 01:22:02.343386 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.343164    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="a24c7c0a-782b-4d04-976d-9f62a964250d" containerName="istio-proxy"
Oct 31 01:22:02.343386 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.343187    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="c975bedf-cccf-4102-bd06-d62598138f5d" containerName="payload-pre-processing"
Oct 31 01:22:02.348784 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.348756    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.378774 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.378735    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8f0029-openshift-default-dockercfg-g7gfd\""
Oct 31 01:22:02.383124 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.383088    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"]
Oct 31 01:22:02.394833 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.394799    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.395013 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.394847    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.395013 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.394954    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.395146 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.395022    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.395146 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.395096    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.395261 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.395188    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgtgl\" (UniqueName: \"kubernetes.io/projected/da54fda6-be77-4967-8c98-8b1f99beeba6-kube-api-access-lgtgl\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.395261 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.395247    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.395367 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.395298    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.395367 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.395335    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da54fda6-be77-4967-8c98-8b1f99beeba6-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.497165 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.497124    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.497370 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.497174    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.497370 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.497198    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.497370 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.497228    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.497370 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.497253    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.497370 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.497306    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgtgl\" (UniqueName: \"kubernetes.io/projected/da54fda6-be77-4967-8c98-8b1f99beeba6-kube-api-access-lgtgl\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.497370 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.497351    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.497823 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.497383    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.497823 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.497422    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da54fda6-be77-4967-8c98-8b1f99beeba6-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.497823 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.497652    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.497823 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.497814    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.497986 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.497891    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.497986 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.497895    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.498166 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.498146    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da54fda6-be77-4967-8c98-8b1f99beeba6-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.499513 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.499495    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.500094 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.500074    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.556118 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.556072    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.556287 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.556231    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgtgl\" (UniqueName: \"kubernetes.io/projected/da54fda6-be77-4967-8c98-8b1f99beeba6-kube-api-access-lgtgl\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.663546 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.663510    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:02.983604 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:22:02.983570    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda54fda6_be77_4967_8c98_8b1f99beeba6.slice/crio-aec4500db618022927378ac134204660ea3f41cbff0b4d7a59c8848036eab154 WatchSource:0}: Error finding container aec4500db618022927378ac134204660ea3f41cbff0b4d7a59c8848036eab154: Status 404 returned error can't find the container with id aec4500db618022927378ac134204660ea3f41cbff0b4d7a59c8848036eab154
Oct 31 01:22:02.984721 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.984660    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"]
Oct 31 01:22:02.986351 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.986308    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:22:02.986464 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.986393    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:22:02.986464 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:02.986440    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:22:03.015123 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:03.015092    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q" event={"ID":"da54fda6-be77-4967-8c98-8b1f99beeba6","Type":"ContainerStarted","Data":"aec4500db618022927378ac134204660ea3f41cbff0b4d7a59c8848036eab154"}
Oct 31 01:22:03.455590 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:03.455554    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl"]
Oct 31 01:22:03.537446 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:03.537410    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x"]
Oct 31 01:22:04.066720 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:04.066638    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q" event={"ID":"da54fda6-be77-4967-8c98-8b1f99beeba6","Type":"ContainerStarted","Data":"e8b38a3ee69bf4a5efbae4bf28e355f7da6cc6b93dfffe6c42f60a9e8119bf67"}
Oct 31 01:22:04.136136 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:04.136069    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q" podStartSLOduration=2.1360459609999998 podStartE2EDuration="2.136045961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:04.111667404 +0000 UTC m=+3784.333702974" watchObservedRunningTime="2026-10-31 01:22:04.136045961 +0000 UTC m=+3784.358081503"
Oct 31 01:22:04.663638 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:04.663600    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:04.665262 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:04.665233    4327 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 31 01:22:04.665400 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:04.665280    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q" podUID="da54fda6-be77-4967-8c98-8b1f99beeba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 31 01:22:05.374123 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:05.374082    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx" podUID="eab68ddb-030b-40ba-a1f9-bac24591237e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:22:05.664154 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:05.664067    4327 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 31 01:22:05.664154 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:05.664126    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q" podUID="da54fda6-be77-4967-8c98-8b1f99beeba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 31 01:22:06.664444 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:06.664400    4327 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 31 01:22:06.665122 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:06.665081    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q" podUID="da54fda6-be77-4967-8c98-8b1f99beeba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 31 01:22:06.739783 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:06.739080    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz" podUID="08107827-859a-452d-9d7e-85377ee4b3ab" containerName="main" containerID="cri-o://6e019a6222b063cbcb151b19cd61c80d86999c1ac711672cca9b7270f9d220e8" gracePeriod=45
Oct 31 01:22:07.086215 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.086182    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"
Oct 31 01:22:07.088626 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.088594    4327 generic.go:356] "Generic (PLEG): container finished" podID="08107827-859a-452d-9d7e-85377ee4b3ab" containerID="6e019a6222b063cbcb151b19cd61c80d86999c1ac711672cca9b7270f9d220e8" exitCode=0
Oct 31 01:22:07.088802 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.088632    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"
Oct 31 01:22:07.088802 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.088670    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz" event={"ID":"08107827-859a-452d-9d7e-85377ee4b3ab","Type":"ContainerDied","Data":"6e019a6222b063cbcb151b19cd61c80d86999c1ac711672cca9b7270f9d220e8"}
Oct 31 01:22:07.088802 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.088768    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz" event={"ID":"08107827-859a-452d-9d7e-85377ee4b3ab","Type":"ContainerDied","Data":"a25c4b7470046275db384eb29fcc7183d6ce551fd9c959c02c8099f8e190d73c"}
Oct 31 01:22:07.088802 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.088790    4327 scope.go:122] "RemoveContainer" containerID="6e019a6222b063cbcb151b19cd61c80d86999c1ac711672cca9b7270f9d220e8"
Oct 31 01:22:07.100319 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.100292    4327 scope.go:122] "RemoveContainer" containerID="6e019a6222b063cbcb151b19cd61c80d86999c1ac711672cca9b7270f9d220e8"
Oct 31 01:22:07.100595 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:22:07.100570    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e019a6222b063cbcb151b19cd61c80d86999c1ac711672cca9b7270f9d220e8\": container with ID starting with 6e019a6222b063cbcb151b19cd61c80d86999c1ac711672cca9b7270f9d220e8 not found: ID does not exist" containerID="6e019a6222b063cbcb151b19cd61c80d86999c1ac711672cca9b7270f9d220e8"
Oct 31 01:22:07.100664 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.100608    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e019a6222b063cbcb151b19cd61c80d86999c1ac711672cca9b7270f9d220e8"} err="failed to get container status \"6e019a6222b063cbcb151b19cd61c80d86999c1ac711672cca9b7270f9d220e8\": rpc error: code = NotFound desc = could not find container \"6e019a6222b063cbcb151b19cd61c80d86999c1ac711672cca9b7270f9d220e8\": container with ID starting with 6e019a6222b063cbcb151b19cd61c80d86999c1ac711672cca9b7270f9d220e8 not found: ID does not exist"
Oct 31 01:22:07.205035 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.205005    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08107827-859a-452d-9d7e-85377ee4b3ab-tls-certs\" (UniqueName: \"kubernetes.io/secret/08107827-859a-452d-9d7e-85377ee4b3ab-tls-certs\") pod \"08107827-859a-452d-9d7e-85377ee4b3ab\" (UID: \"08107827-859a-452d-9d7e-85377ee4b3ab\") "
Oct 31 01:22:07.205220 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.205095    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-tmp-dir\") pod \"08107827-859a-452d-9d7e-85377ee4b3ab\" (UID: \"08107827-859a-452d-9d7e-85377ee4b3ab\") "
Oct 31 01:22:07.205220 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.205146    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-model-cache\") pod \"08107827-859a-452d-9d7e-85377ee4b3ab\" (UID: \"08107827-859a-452d-9d7e-85377ee4b3ab\") "
Oct 31 01:22:07.205220 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.205169    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-dshm\" (UniqueName: \"kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-dshm\") pod \"08107827-859a-452d-9d7e-85377ee4b3ab\" (UID: \"08107827-859a-452d-9d7e-85377ee4b3ab\") "
Oct 31 01:22:07.205220 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.205201    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-home\" (UniqueName: \"kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-home\") pod \"08107827-859a-452d-9d7e-85377ee4b3ab\" (UID: \"08107827-859a-452d-9d7e-85377ee4b3ab\") "
Oct 31 01:22:07.205502 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.205425    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-tmp-dir" pod "08107827-859a-452d-9d7e-85377ee4b3ab" (UID: "08107827-859a-452d-9d7e-85377ee4b3ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:22:07.205502 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.205449    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-model-cache" pod "08107827-859a-452d-9d7e-85377ee4b3ab" (UID: "08107827-859a-452d-9d7e-85377ee4b3ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:22:07.205688 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.205505    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-home" pod "08107827-859a-452d-9d7e-85377ee4b3ab" (UID: "08107827-859a-452d-9d7e-85377ee4b3ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:22:07.205737 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.205713    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08107827-859a-452d-9d7e-85377ee4b3ab-kube-api-access-gnrst\" (UniqueName: \"kubernetes.io/projected/08107827-859a-452d-9d7e-85377ee4b3ab-kube-api-access-gnrst\") pod \"08107827-859a-452d-9d7e-85377ee4b3ab\" (UID: \"08107827-859a-452d-9d7e-85377ee4b3ab\") "
Oct 31 01:22:07.206360 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.206336    4327 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-home\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:07.206360 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.206360    4327 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-tmp-dir\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:07.206537 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.206370    4327 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-model-cache\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:07.207461 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.207436    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08107827-859a-452d-9d7e-85377ee4b3ab-tls-certs" pod "08107827-859a-452d-9d7e-85377ee4b3ab" (UID: "08107827-859a-452d-9d7e-85377ee4b3ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:22:07.207580 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.207558    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-dshm" pod "08107827-859a-452d-9d7e-85377ee4b3ab" (UID: "08107827-859a-452d-9d7e-85377ee4b3ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:22:07.207764 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.207737    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08107827-859a-452d-9d7e-85377ee4b3ab-kube-api-access-gnrst" pod "08107827-859a-452d-9d7e-85377ee4b3ab" (UID: "08107827-859a-452d-9d7e-85377ee4b3ab"). InnerVolumeSpecName "kube-api-access-gnrst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:22:07.307996 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.307951    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnrst\" (UniqueName: \"kubernetes.io/projected/08107827-859a-452d-9d7e-85377ee4b3ab-kube-api-access-gnrst\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:07.307996 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.307990    4327 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08107827-859a-452d-9d7e-85377ee4b3ab-tls-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:07.307996 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.307999    4327 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08107827-859a-452d-9d7e-85377ee4b3ab-dshm\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:07.443997 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.443947    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"]
Oct 31 01:22:07.447724 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.447501    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ada0c1/rate-test-model-ada0c1-kserve-5985cf6565-qslxz"]
Oct 31 01:22:07.664106 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.664070    4327 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 31 01:22:07.664282 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.664122    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q" podUID="da54fda6-be77-4967-8c98-8b1f99beeba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 31 01:22:07.942906 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:07.942823    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08107827-859a-452d-9d7e-85377ee4b3ab" path="/var/lib/kubelet/pods/08107827-859a-452d-9d7e-85377ee4b3ab/volumes"
Oct 31 01:22:08.358989 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:08.357590    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"]
Oct 31 01:22:08.358989 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:08.357896    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh" podUID="a811bb1a-42b1-4184-b123-c295547116d2" containerName="istio-proxy" containerID="cri-o://4e46f7b1f6bd74edd63f973e1001089653201cceaad28c7ebf1dbfd8abdf0bb4" gracePeriod=30
Oct 31 01:22:08.460290 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:08.458763    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl" podUID="7549bb85-c79e-4740-b3f0-7f1c793b741a" containerName="payload-processing" containerID="cri-o://5347aaec0f96079409844a760560609fe8056c79804deb47d9c00869cc00df49" gracePeriod=25
Oct 31 01:22:08.550699 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:08.549796    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x" podUID="b835588e-1042-46bb-8554-243c7dab259d" containerName="payload-pre-processing" containerID="cri-o://4140e3d7f9077107def2088220d601f2300748e23e1d547223716fd99eabc10d" gracePeriod=25
Oct 31 01:22:08.665034 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:08.664952    4327 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 31 01:22:08.665034 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:08.665012    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q" podUID="da54fda6-be77-4967-8c98-8b1f99beeba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 31 01:22:08.728701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:08.716803    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx"
Oct 31 01:22:08.859775 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:08.858366    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eab68ddb-030b-40ba-a1f9-bac24591237e-kube-api-access-wpt59\" (UniqueName: \"kubernetes.io/projected/eab68ddb-030b-40ba-a1f9-bac24591237e-kube-api-access-wpt59\") pod \"eab68ddb-030b-40ba-a1f9-bac24591237e\" (UID: \"eab68ddb-030b-40ba-a1f9-bac24591237e\") "
Oct 31 01:22:08.859775 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:08.858621    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eab68ddb-030b-40ba-a1f9-bac24591237e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab68ddb-030b-40ba-a1f9-bac24591237e-plugins-config-volume\") pod \"eab68ddb-030b-40ba-a1f9-bac24591237e\" (UID: \"eab68ddb-030b-40ba-a1f9-bac24591237e\") "
Oct 31 01:22:08.859775 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:08.859372    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eab68ddb-030b-40ba-a1f9-bac24591237e-plugins-config-volume" pod "eab68ddb-030b-40ba-a1f9-bac24591237e" (UID: "eab68ddb-030b-40ba-a1f9-bac24591237e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:22:08.860908 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:08.860885    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eab68ddb-030b-40ba-a1f9-bac24591237e-kube-api-access-wpt59" pod "eab68ddb-030b-40ba-a1f9-bac24591237e" (UID: "eab68ddb-030b-40ba-a1f9-bac24591237e"). InnerVolumeSpecName "kube-api-access-wpt59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:22:08.961059 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:08.960944    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab68ddb-030b-40ba-a1f9-bac24591237e-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:08.961059 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:08.960980    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpt59\" (UniqueName: \"kubernetes.io/projected/eab68ddb-030b-40ba-a1f9-bac24591237e-kube-api-access-wpt59\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:09.141772 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:09.116205    4327 generic.go:356] "Generic (PLEG): container finished" podID="eab68ddb-030b-40ba-a1f9-bac24591237e" containerID="ab8a8b3f59b553f92097ccd7ebc677c631e39eefa02ce1c0e536f713408ed119" exitCode=0
Oct 31 01:22:09.141772 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:09.116324    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx" event={"ID":"eab68ddb-030b-40ba-a1f9-bac24591237e","Type":"ContainerDied","Data":"ab8a8b3f59b553f92097ccd7ebc677c631e39eefa02ce1c0e536f713408ed119"}
Oct 31 01:22:09.141772 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:09.116410    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx" event={"ID":"eab68ddb-030b-40ba-a1f9-bac24591237e","Type":"ContainerDied","Data":"c859b33227dea07397469719d490536b7d0d5a5fe95df21e7fc33ddb8248a09b"}
Oct 31 01:22:09.141772 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:09.116441    4327 scope.go:122] "RemoveContainer" containerID="ab8a8b3f59b553f92097ccd7ebc677c631e39eefa02ce1c0e536f713408ed119"
Oct 31 01:22:09.141772 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:09.116601    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx"
Oct 31 01:22:09.159966 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:09.155417    4327 scope.go:122] "RemoveContainer" containerID="ab8a8b3f59b553f92097ccd7ebc677c631e39eefa02ce1c0e536f713408ed119"
Oct 31 01:22:09.159966 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:22:09.155798    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab8a8b3f59b553f92097ccd7ebc677c631e39eefa02ce1c0e536f713408ed119\": container with ID starting with ab8a8b3f59b553f92097ccd7ebc677c631e39eefa02ce1c0e536f713408ed119 not found: ID does not exist" containerID="ab8a8b3f59b553f92097ccd7ebc677c631e39eefa02ce1c0e536f713408ed119"
Oct 31 01:22:09.159966 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:09.155829    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab8a8b3f59b553f92097ccd7ebc677c631e39eefa02ce1c0e536f713408ed119"} err="failed to get container status \"ab8a8b3f59b553f92097ccd7ebc677c631e39eefa02ce1c0e536f713408ed119\": rpc error: code = NotFound desc = could not find container \"ab8a8b3f59b553f92097ccd7ebc677c631e39eefa02ce1c0e536f713408ed119\": container with ID starting with ab8a8b3f59b553f92097ccd7ebc677c631e39eefa02ce1c0e536f713408ed119 not found: ID does not exist"
Oct 31 01:22:09.198770 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:09.198716    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx"]
Oct 31 01:22:09.201965 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:09.201939    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-714f8f-694d787d6c-4tbbx"]
Oct 31 01:22:09.672947 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:09.672915    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:09.673314 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:09.673289    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:09.674422 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:09.674398    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:22:09.922014 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:09.921983    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eab68ddb-030b-40ba-a1f9-bac24591237e" path="/var/lib/kubelet/pods/eab68ddb-030b-40ba-a1f9-bac24591237e/volumes"
Oct 31 01:22:13.133473 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.133429    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:22:13.133904 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.133516    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh" podUID="a811bb1a-42b1-4184-b123-c295547116d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:22:13.628292 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.628265    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 01:22:13.725640 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.725551    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-credential-socket\") pod \"a811bb1a-42b1-4184-b123-c295547116d2\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") "
Oct 31 01:22:13.725640 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.725614    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a811bb1a-42b1-4184-b123-c295547116d2-kube-api-access-hhpdx\" (UniqueName: \"kubernetes.io/projected/a811bb1a-42b1-4184-b123-c295547116d2-kube-api-access-hhpdx\") pod \"a811bb1a-42b1-4184-b123-c295547116d2\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") "
Oct 31 01:22:13.725927 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.725670    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a811bb1a-42b1-4184-b123-c295547116d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a811bb1a-42b1-4184-b123-c295547116d2-istiod-ca-cert\") pod \"a811bb1a-42b1-4184-b123-c295547116d2\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") "
Oct 31 01:22:13.725927 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.725723    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a811bb1a-42b1-4184-b123-c295547116d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a811bb1a-42b1-4184-b123-c295547116d2-istio-podinfo\") pod \"a811bb1a-42b1-4184-b123-c295547116d2\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") "
Oct 31 01:22:13.725927 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.725890    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-credential-socket" pod "a811bb1a-42b1-4184-b123-c295547116d2" (UID: "a811bb1a-42b1-4184-b123-c295547116d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:22:13.726106 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.725990    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-workload-socket\") pod \"a811bb1a-42b1-4184-b123-c295547116d2\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") "
Oct 31 01:22:13.726171 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.726096    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a811bb1a-42b1-4184-b123-c295547116d2-istiod-ca-cert" pod "a811bb1a-42b1-4184-b123-c295547116d2" (UID: "a811bb1a-42b1-4184-b123-c295547116d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:22:13.726227 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.726191    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-workload-socket" pod "a811bb1a-42b1-4184-b123-c295547116d2" (UID: "a811bb1a-42b1-4184-b123-c295547116d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:22:13.726336 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.726313    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-workload-certs\") pod \"a811bb1a-42b1-4184-b123-c295547116d2\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") "
Oct 31 01:22:13.726411 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.726381    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-istio-envoy\") pod \"a811bb1a-42b1-4184-b123-c295547116d2\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") "
Oct 31 01:22:13.726471 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.726425    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a811bb1a-42b1-4184-b123-c295547116d2-istio-token\" (UniqueName: \"kubernetes.io/projected/a811bb1a-42b1-4184-b123-c295547116d2-istio-token\") pod \"a811bb1a-42b1-4184-b123-c295547116d2\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") "
Oct 31 01:22:13.726534 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.726497    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-istio-data\") pod \"a811bb1a-42b1-4184-b123-c295547116d2\" (UID: \"a811bb1a-42b1-4184-b123-c295547116d2\") "
Oct 31 01:22:13.726612 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.726589    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-workload-certs" pod "a811bb1a-42b1-4184-b123-c295547116d2" (UID: "a811bb1a-42b1-4184-b123-c295547116d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:22:13.727040 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.726995    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-istio-data" pod "a811bb1a-42b1-4184-b123-c295547116d2" (UID: "a811bb1a-42b1-4184-b123-c295547116d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:22:13.727163 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.727052    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:13.727163 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.727074    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:13.727163 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.727088    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a811bb1a-42b1-4184-b123-c295547116d2-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:13.727163 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.727102    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:13.729191 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.729162    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-istio-envoy" pod "a811bb1a-42b1-4184-b123-c295547116d2" (UID: "a811bb1a-42b1-4184-b123-c295547116d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:22:13.729191 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.729179    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a811bb1a-42b1-4184-b123-c295547116d2-istio-podinfo" pod "a811bb1a-42b1-4184-b123-c295547116d2" (UID: "a811bb1a-42b1-4184-b123-c295547116d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:22:13.729341 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.729193    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a811bb1a-42b1-4184-b123-c295547116d2-istio-token" pod "a811bb1a-42b1-4184-b123-c295547116d2" (UID: "a811bb1a-42b1-4184-b123-c295547116d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:22:13.729408 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.729390    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a811bb1a-42b1-4184-b123-c295547116d2-kube-api-access-hhpdx" pod "a811bb1a-42b1-4184-b123-c295547116d2" (UID: "a811bb1a-42b1-4184-b123-c295547116d2"). InnerVolumeSpecName "kube-api-access-hhpdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:22:13.828734 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.828693    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:13.828734 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.828725    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a811bb1a-42b1-4184-b123-c295547116d2-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:13.828734 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.828738    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a811bb1a-42b1-4184-b123-c295547116d2-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:13.829016 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.828753    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhpdx\" (UniqueName: \"kubernetes.io/projected/a811bb1a-42b1-4184-b123-c295547116d2-kube-api-access-hhpdx\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:13.829016 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:13.828766    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a811bb1a-42b1-4184-b123-c295547116d2-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:14.151003 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:14.150968    4327 generic.go:356] "Generic (PLEG): container finished" podID="a811bb1a-42b1-4184-b123-c295547116d2" containerID="4e46f7b1f6bd74edd63f973e1001089653201cceaad28c7ebf1dbfd8abdf0bb4" exitCode=0
Oct 31 01:22:14.151421 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:14.151018    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"
Oct 31 01:22:14.151421 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:14.151054    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh" event={"ID":"a811bb1a-42b1-4184-b123-c295547116d2","Type":"ContainerDied","Data":"4e46f7b1f6bd74edd63f973e1001089653201cceaad28c7ebf1dbfd8abdf0bb4"}
Oct 31 01:22:14.151421 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:14.151119    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh" event={"ID":"a811bb1a-42b1-4184-b123-c295547116d2","Type":"ContainerDied","Data":"16b5cabb7c6178ed9cbeffb17db136fd771cd5813336543f26cb5ce9c956f77c"}
Oct 31 01:22:14.151421 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:14.151144    4327 scope.go:122] "RemoveContainer" containerID="4e46f7b1f6bd74edd63f973e1001089653201cceaad28c7ebf1dbfd8abdf0bb4"
Oct 31 01:22:14.161105 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:14.161086    4327 scope.go:122] "RemoveContainer" containerID="8d72e1c71501630b0c3c7adcfd48ba45fda3046ef9b961815a15a8b58b5adf13"
Oct 31 01:22:14.170319 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:14.170294    4327 scope.go:122] "RemoveContainer" containerID="4e46f7b1f6bd74edd63f973e1001089653201cceaad28c7ebf1dbfd8abdf0bb4"
Oct 31 01:22:14.170654 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:22:14.170632    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e46f7b1f6bd74edd63f973e1001089653201cceaad28c7ebf1dbfd8abdf0bb4\": container with ID starting with 4e46f7b1f6bd74edd63f973e1001089653201cceaad28c7ebf1dbfd8abdf0bb4 not found: ID does not exist" containerID="4e46f7b1f6bd74edd63f973e1001089653201cceaad28c7ebf1dbfd8abdf0bb4"
Oct 31 01:22:14.170762 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:14.170665    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e46f7b1f6bd74edd63f973e1001089653201cceaad28c7ebf1dbfd8abdf0bb4"} err="failed to get container status \"4e46f7b1f6bd74edd63f973e1001089653201cceaad28c7ebf1dbfd8abdf0bb4\": rpc error: code = NotFound desc = could not find container \"4e46f7b1f6bd74edd63f973e1001089653201cceaad28c7ebf1dbfd8abdf0bb4\": container with ID starting with 4e46f7b1f6bd74edd63f973e1001089653201cceaad28c7ebf1dbfd8abdf0bb4 not found: ID does not exist"
Oct 31 01:22:14.170762 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:14.170703    4327 scope.go:122] "RemoveContainer" containerID="8d72e1c71501630b0c3c7adcfd48ba45fda3046ef9b961815a15a8b58b5adf13"
Oct 31 01:22:14.170979 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:22:14.170961    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d72e1c71501630b0c3c7adcfd48ba45fda3046ef9b961815a15a8b58b5adf13\": container with ID starting with 8d72e1c71501630b0c3c7adcfd48ba45fda3046ef9b961815a15a8b58b5adf13 not found: ID does not exist" containerID="8d72e1c71501630b0c3c7adcfd48ba45fda3046ef9b961815a15a8b58b5adf13"
Oct 31 01:22:14.171035 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:14.170983    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d72e1c71501630b0c3c7adcfd48ba45fda3046ef9b961815a15a8b58b5adf13"} err="failed to get container status \"8d72e1c71501630b0c3c7adcfd48ba45fda3046ef9b961815a15a8b58b5adf13\": rpc error: code = NotFound desc = could not find container \"8d72e1c71501630b0c3c7adcfd48ba45fda3046ef9b961815a15a8b58b5adf13\": container with ID starting with 8d72e1c71501630b0c3c7adcfd48ba45fda3046ef9b961815a15a8b58b5adf13 not found: ID does not exist"
Oct 31 01:22:14.204308 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:14.204277    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"]
Oct 31 01:22:14.214069 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:14.214041    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-nmddh"]
Oct 31 01:22:15.278884 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:15.278848    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl" podUID="7549bb85-c79e-4740-b3f0-7f1c793b741a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:22:15.278884 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:15.278864    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x" podUID="b835588e-1042-46bb-8554-243c7dab259d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:22:15.293949 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:22:15.293912    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:22:15.921583 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:15.921554    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a811bb1a-42b1-4184-b123-c295547116d2" path="/var/lib/kubelet/pods/a811bb1a-42b1-4184-b123-c295547116d2/volumes"
Oct 31 01:22:25.279806 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:25.279766    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x" podUID="b835588e-1042-46bb-8554-243c7dab259d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:22:25.279806 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:25.279791    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl" podUID="7549bb85-c79e-4740-b3f0-7f1c793b741a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:22:28.840881 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:28.840848    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl"
Oct 31 01:22:28.878981 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:28.878956    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x"
Oct 31 01:22:28.913199 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:28.913162    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7549bb85-c79e-4740-b3f0-7f1c793b741a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7549bb85-c79e-4740-b3f0-7f1c793b741a-plugins-config-volume\") pod \"7549bb85-c79e-4740-b3f0-7f1c793b741a\" (UID: \"7549bb85-c79e-4740-b3f0-7f1c793b741a\") "
Oct 31 01:22:28.913554 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:28.913530    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7549bb85-c79e-4740-b3f0-7f1c793b741a-kube-api-access-d9mkf\" (UniqueName: \"kubernetes.io/projected/7549bb85-c79e-4740-b3f0-7f1c793b741a-kube-api-access-d9mkf\") pod \"7549bb85-c79e-4740-b3f0-7f1c793b741a\" (UID: \"7549bb85-c79e-4740-b3f0-7f1c793b741a\") "
Oct 31 01:22:28.916951 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:28.916921    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7549bb85-c79e-4740-b3f0-7f1c793b741a-plugins-config-volume" pod "7549bb85-c79e-4740-b3f0-7f1c793b741a" (UID: "7549bb85-c79e-4740-b3f0-7f1c793b741a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:22:28.926645 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:28.926615    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7549bb85-c79e-4740-b3f0-7f1c793b741a-kube-api-access-d9mkf" pod "7549bb85-c79e-4740-b3f0-7f1c793b741a" (UID: "7549bb85-c79e-4740-b3f0-7f1c793b741a"). InnerVolumeSpecName "kube-api-access-d9mkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:22:29.014869 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.014834    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b835588e-1042-46bb-8554-243c7dab259d-kube-api-access-kf6px\" (UniqueName: \"kubernetes.io/projected/b835588e-1042-46bb-8554-243c7dab259d-kube-api-access-kf6px\") pod \"b835588e-1042-46bb-8554-243c7dab259d\" (UID: \"b835588e-1042-46bb-8554-243c7dab259d\") "
Oct 31 01:22:29.015063 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.014947    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b835588e-1042-46bb-8554-243c7dab259d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b835588e-1042-46bb-8554-243c7dab259d-plugins-config-volume\") pod \"b835588e-1042-46bb-8554-243c7dab259d\" (UID: \"b835588e-1042-46bb-8554-243c7dab259d\") "
Oct 31 01:22:29.015330 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.015308    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b835588e-1042-46bb-8554-243c7dab259d-plugins-config-volume" pod "b835588e-1042-46bb-8554-243c7dab259d" (UID: "b835588e-1042-46bb-8554-243c7dab259d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:22:29.015467 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.015448    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b835588e-1042-46bb-8554-243c7dab259d-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:29.015540 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.015470    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7549bb85-c79e-4740-b3f0-7f1c793b741a-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:29.015540 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.015479    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9mkf\" (UniqueName: \"kubernetes.io/projected/7549bb85-c79e-4740-b3f0-7f1c793b741a-kube-api-access-d9mkf\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:29.017131 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.017105    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b835588e-1042-46bb-8554-243c7dab259d-kube-api-access-kf6px" pod "b835588e-1042-46bb-8554-243c7dab259d" (UID: "b835588e-1042-46bb-8554-243c7dab259d"). InnerVolumeSpecName "kube-api-access-kf6px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:22:29.116423 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.116389    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf6px\" (UniqueName: \"kubernetes.io/projected/b835588e-1042-46bb-8554-243c7dab259d-kube-api-access-kf6px\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:29.264581 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.264542    4327 generic.go:356] "Generic (PLEG): container finished" podID="7549bb85-c79e-4740-b3f0-7f1c793b741a" containerID="5347aaec0f96079409844a760560609fe8056c79804deb47d9c00869cc00df49" exitCode=0
Oct 31 01:22:29.264823 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.264662    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl" event={"ID":"7549bb85-c79e-4740-b3f0-7f1c793b741a","Type":"ContainerDied","Data":"5347aaec0f96079409844a760560609fe8056c79804deb47d9c00869cc00df49"}
Oct 31 01:22:29.264823 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.264775    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl" event={"ID":"7549bb85-c79e-4740-b3f0-7f1c793b741a","Type":"ContainerDied","Data":"932e4819b18acf08d65834b52445671375661732c9426b152f906e334e26f1fd"}
Oct 31 01:22:29.264823 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.264701    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl"
Oct 31 01:22:29.265008 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.264811    4327 scope.go:122] "RemoveContainer" containerID="5347aaec0f96079409844a760560609fe8056c79804deb47d9c00869cc00df49"
Oct 31 01:22:29.266707 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.266574    4327 generic.go:356] "Generic (PLEG): container finished" podID="b835588e-1042-46bb-8554-243c7dab259d" containerID="4140e3d7f9077107def2088220d601f2300748e23e1d547223716fd99eabc10d" exitCode=0
Oct 31 01:22:29.266844 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.266819    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x" event={"ID":"b835588e-1042-46bb-8554-243c7dab259d","Type":"ContainerDied","Data":"4140e3d7f9077107def2088220d601f2300748e23e1d547223716fd99eabc10d"}
Oct 31 01:22:29.267055 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.266868    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x" event={"ID":"b835588e-1042-46bb-8554-243c7dab259d","Type":"ContainerDied","Data":"c2275dc6566b03da1469973109a2ce6b44529e9e1115d972e505c9d2fda64960"}
Oct 31 01:22:29.267055 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.266951    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x"
Oct 31 01:22:29.278151 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.278130    4327 scope.go:122] "RemoveContainer" containerID="5347aaec0f96079409844a760560609fe8056c79804deb47d9c00869cc00df49"
Oct 31 01:22:29.278585 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:22:29.278550    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5347aaec0f96079409844a760560609fe8056c79804deb47d9c00869cc00df49\": container with ID starting with 5347aaec0f96079409844a760560609fe8056c79804deb47d9c00869cc00df49 not found: ID does not exist" containerID="5347aaec0f96079409844a760560609fe8056c79804deb47d9c00869cc00df49"
Oct 31 01:22:29.278742 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.278595    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5347aaec0f96079409844a760560609fe8056c79804deb47d9c00869cc00df49"} err="failed to get container status \"5347aaec0f96079409844a760560609fe8056c79804deb47d9c00869cc00df49\": rpc error: code = NotFound desc = could not find container \"5347aaec0f96079409844a760560609fe8056c79804deb47d9c00869cc00df49\": container with ID starting with 5347aaec0f96079409844a760560609fe8056c79804deb47d9c00869cc00df49 not found: ID does not exist"
Oct 31 01:22:29.278742 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.278617    4327 scope.go:122] "RemoveContainer" containerID="4140e3d7f9077107def2088220d601f2300748e23e1d547223716fd99eabc10d"
Oct 31 01:22:29.291655 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.291634    4327 scope.go:122] "RemoveContainer" containerID="4140e3d7f9077107def2088220d601f2300748e23e1d547223716fd99eabc10d"
Oct 31 01:22:29.291983 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:22:29.291959    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4140e3d7f9077107def2088220d601f2300748e23e1d547223716fd99eabc10d\": container with ID starting with 4140e3d7f9077107def2088220d601f2300748e23e1d547223716fd99eabc10d not found: ID does not exist" containerID="4140e3d7f9077107def2088220d601f2300748e23e1d547223716fd99eabc10d"
Oct 31 01:22:29.292086 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.291990    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4140e3d7f9077107def2088220d601f2300748e23e1d547223716fd99eabc10d"} err="failed to get container status \"4140e3d7f9077107def2088220d601f2300748e23e1d547223716fd99eabc10d\": rpc error: code = NotFound desc = could not find container \"4140e3d7f9077107def2088220d601f2300748e23e1d547223716fd99eabc10d\": container with ID starting with 4140e3d7f9077107def2088220d601f2300748e23e1d547223716fd99eabc10d not found: ID does not exist"
Oct 31 01:22:29.336900 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.336870    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x"]
Oct 31 01:22:29.349216 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.349187    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d10721-6d9947dc59-xfh7x"]
Oct 31 01:22:29.383803 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.383728    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl"]
Oct 31 01:22:29.388851 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.388824    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d10721-7cb984c4cc-jtcsl"]
Oct 31 01:22:29.921383 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.921347    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7549bb85-c79e-4740-b3f0-7f1c793b741a" path="/var/lib/kubelet/pods/7549bb85-c79e-4740-b3f0-7f1c793b741a/volumes"
Oct 31 01:22:29.921971 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:29.921953    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b835588e-1042-46bb-8554-243c7dab259d" path="/var/lib/kubelet/pods/b835588e-1042-46bb-8554-243c7dab259d/volumes"
Oct 31 01:22:35.890902 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:35.890864    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"]
Oct 31 01:22:35.891409 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:35.891172    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7" podUID="f745fa94-dca6-46ac-981e-73ce0831e867" containerName="istio-proxy" containerID="cri-o://8bb830716b53d48692fb82190c7393afdd08a6fb0f43696b85306ffc25db8f54" gracePeriod=30
Oct 31 01:22:41.161427 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.161395    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr"]
Oct 31 01:22:41.161933 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.161654    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr" podUID="fdeebb2d-1d8b-49e0-bdee-af282e78e7a1" containerName="payload-processing" containerID="cri-o://d1238ead261897c88a71f624d675cec283b8ebd3e15f78a4fee71b3978a69177" gracePeriod=30
Oct 31 01:22:41.194240 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.194206    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd"]
Oct 31 01:22:41.194500 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.194470    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd" podUID="e60feb74-e154-496d-88d1-9e8902b2e7ae" containerName="payload-pre-processing" containerID="cri-o://52d3a2d5f329673403c5f94f5fbb577785af0cb07ff7d84b54fab6cb3c29dc84" gracePeriod=30
Oct 31 01:22:41.319974 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.319947    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:22:41.347724 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.347615    4327 generic.go:356] "Generic (PLEG): container finished" podID="fdeebb2d-1d8b-49e0-bdee-af282e78e7a1" containerID="d1238ead261897c88a71f624d675cec283b8ebd3e15f78a4fee71b3978a69177" exitCode=0
Oct 31 01:22:41.347942 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.347797    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr" event={"ID":"fdeebb2d-1d8b-49e0-bdee-af282e78e7a1","Type":"ContainerDied","Data":"d1238ead261897c88a71f624d675cec283b8ebd3e15f78a4fee71b3978a69177"}
Oct 31 01:22:41.349568 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.349533    4327 generic.go:356] "Generic (PLEG): container finished" podID="f745fa94-dca6-46ac-981e-73ce0831e867" containerID="8bb830716b53d48692fb82190c7393afdd08a6fb0f43696b85306ffc25db8f54" exitCode=0
Oct 31 01:22:41.349754 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.349729    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7" event={"ID":"f745fa94-dca6-46ac-981e-73ce0831e867","Type":"ContainerDied","Data":"8bb830716b53d48692fb82190c7393afdd08a6fb0f43696b85306ffc25db8f54"}
Oct 31 01:22:41.349836 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.349781    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7" event={"ID":"f745fa94-dca6-46ac-981e-73ce0831e867","Type":"ContainerDied","Data":"f31068e2d23c0c0cb31f41087c319f4eb969c0069bc67420f7024e83f280d542"}
Oct 31 01:22:41.349836 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.349815    4327 scope.go:122] "RemoveContainer" containerID="8bb830716b53d48692fb82190c7393afdd08a6fb0f43696b85306ffc25db8f54"
Oct 31 01:22:41.349985 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.349970    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"
Oct 31 01:22:41.352457 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.352433    4327 generic.go:356] "Generic (PLEG): container finished" podID="e60feb74-e154-496d-88d1-9e8902b2e7ae" containerID="52d3a2d5f329673403c5f94f5fbb577785af0cb07ff7d84b54fab6cb3c29dc84" exitCode=0
Oct 31 01:22:41.352576 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.352500    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd" event={"ID":"e60feb74-e154-496d-88d1-9e8902b2e7ae","Type":"ContainerDied","Data":"52d3a2d5f329673403c5f94f5fbb577785af0cb07ff7d84b54fab6cb3c29dc84"}
Oct 31 01:22:41.365972 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.365931    4327 scope.go:122] "RemoveContainer" containerID="8bb830716b53d48692fb82190c7393afdd08a6fb0f43696b85306ffc25db8f54"
Oct 31 01:22:41.366415 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:22:41.366381    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bb830716b53d48692fb82190c7393afdd08a6fb0f43696b85306ffc25db8f54\": container with ID starting with 8bb830716b53d48692fb82190c7393afdd08a6fb0f43696b85306ffc25db8f54 not found: ID does not exist" containerID="8bb830716b53d48692fb82190c7393afdd08a6fb0f43696b85306ffc25db8f54"
Oct 31 01:22:41.366535 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.366424    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bb830716b53d48692fb82190c7393afdd08a6fb0f43696b85306ffc25db8f54"} err="failed to get container status \"8bb830716b53d48692fb82190c7393afdd08a6fb0f43696b85306ffc25db8f54\": rpc error: code = NotFound desc = could not find container \"8bb830716b53d48692fb82190c7393afdd08a6fb0f43696b85306ffc25db8f54\": container with ID starting with 8bb830716b53d48692fb82190c7393afdd08a6fb0f43696b85306ffc25db8f54 not found: ID does not exist"
Oct 31 01:22:41.469856 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.469395    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-workload-certs\") pod \"f745fa94-dca6-46ac-981e-73ce0831e867\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") "
Oct 31 01:22:41.469856 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.469446    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f745fa94-dca6-46ac-981e-73ce0831e867-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f745fa94-dca6-46ac-981e-73ce0831e867-istiod-ca-cert\") pod \"f745fa94-dca6-46ac-981e-73ce0831e867\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") "
Oct 31 01:22:41.469856 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.469722    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-workload-certs" pod "f745fa94-dca6-46ac-981e-73ce0831e867" (UID: "f745fa94-dca6-46ac-981e-73ce0831e867"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:22:41.470082 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.470054    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f745fa94-dca6-46ac-981e-73ce0831e867-istiod-ca-cert" pod "f745fa94-dca6-46ac-981e-73ce0831e867" (UID: "f745fa94-dca6-46ac-981e-73ce0831e867"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:22:41.470154 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.470100    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f745fa94-dca6-46ac-981e-73ce0831e867-kube-api-access-lsrpz\" (UniqueName: \"kubernetes.io/projected/f745fa94-dca6-46ac-981e-73ce0831e867-kube-api-access-lsrpz\") pod \"f745fa94-dca6-46ac-981e-73ce0831e867\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") "
Oct 31 01:22:41.470234 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.470213    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr"
Oct 31 01:22:41.471170 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.470418    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-credential-socket" pod "f745fa94-dca6-46ac-981e-73ce0831e867" (UID: "f745fa94-dca6-46ac-981e-73ce0831e867"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:22:41.471170 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.470216    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-credential-socket\") pod \"f745fa94-dca6-46ac-981e-73ce0831e867\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") "
Oct 31 01:22:41.471170 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.470512    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-workload-socket\") pod \"f745fa94-dca6-46ac-981e-73ce0831e867\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") "
Oct 31 01:22:41.471170 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.470584    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f745fa94-dca6-46ac-981e-73ce0831e867-istio-token\" (UniqueName: \"kubernetes.io/projected/f745fa94-dca6-46ac-981e-73ce0831e867-istio-token\") pod \"f745fa94-dca6-46ac-981e-73ce0831e867\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") "
Oct 31 01:22:41.471170 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.470632    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f745fa94-dca6-46ac-981e-73ce0831e867-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f745fa94-dca6-46ac-981e-73ce0831e867-istio-podinfo\") pod \"f745fa94-dca6-46ac-981e-73ce0831e867\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") "
Oct 31 01:22:41.471170 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.470660    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-istio-data\") pod \"f745fa94-dca6-46ac-981e-73ce0831e867\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") "
Oct 31 01:22:41.471170 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.470719    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-istio-envoy\") pod \"f745fa94-dca6-46ac-981e-73ce0831e867\" (UID: \"f745fa94-dca6-46ac-981e-73ce0831e867\") "
Oct 31 01:22:41.471170 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.470866    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-workload-socket" pod "f745fa94-dca6-46ac-981e-73ce0831e867" (UID: "f745fa94-dca6-46ac-981e-73ce0831e867"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:22:41.472114 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.471171    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-istio-data" pod "f745fa94-dca6-46ac-981e-73ce0831e867" (UID: "f745fa94-dca6-46ac-981e-73ce0831e867"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:22:41.472114 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.471893    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:41.472114 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.471918    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:41.472114 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.471935    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:41.472114 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.471950    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:41.472114 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.471963    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f745fa94-dca6-46ac-981e-73ce0831e867-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:41.473941 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.473914    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-istio-envoy" pod "f745fa94-dca6-46ac-981e-73ce0831e867" (UID: "f745fa94-dca6-46ac-981e-73ce0831e867"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:22:41.474871 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.474797    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f745fa94-dca6-46ac-981e-73ce0831e867-istio-token" pod "f745fa94-dca6-46ac-981e-73ce0831e867" (UID: "f745fa94-dca6-46ac-981e-73ce0831e867"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:22:41.476583 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.476561    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f745fa94-dca6-46ac-981e-73ce0831e867-kube-api-access-lsrpz" pod "f745fa94-dca6-46ac-981e-73ce0831e867" (UID: "f745fa94-dca6-46ac-981e-73ce0831e867"). InnerVolumeSpecName "kube-api-access-lsrpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:22:41.476752 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.476724    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f745fa94-dca6-46ac-981e-73ce0831e867-istio-podinfo" pod "f745fa94-dca6-46ac-981e-73ce0831e867" (UID: "f745fa94-dca6-46ac-981e-73ce0831e867"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:22:41.481558 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.481533    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd"
Oct 31 01:22:41.580766 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.573982    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e60feb74-e154-496d-88d1-9e8902b2e7ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e60feb74-e154-496d-88d1-9e8902b2e7ae-plugins-config-volume\") pod \"e60feb74-e154-496d-88d1-9e8902b2e7ae\" (UID: \"e60feb74-e154-496d-88d1-9e8902b2e7ae\") "
Oct 31 01:22:41.580766 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.574059    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdeebb2d-1d8b-49e0-bdee-af282e78e7a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdeebb2d-1d8b-49e0-bdee-af282e78e7a1-plugins-config-volume\") pod \"fdeebb2d-1d8b-49e0-bdee-af282e78e7a1\" (UID: \"fdeebb2d-1d8b-49e0-bdee-af282e78e7a1\") "
Oct 31 01:22:41.580766 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.574129    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdeebb2d-1d8b-49e0-bdee-af282e78e7a1-kube-api-access-zj65s\" (UniqueName: \"kubernetes.io/projected/fdeebb2d-1d8b-49e0-bdee-af282e78e7a1-kube-api-access-zj65s\") pod \"fdeebb2d-1d8b-49e0-bdee-af282e78e7a1\" (UID: \"fdeebb2d-1d8b-49e0-bdee-af282e78e7a1\") "
Oct 31 01:22:41.580766 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.574260    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e60feb74-e154-496d-88d1-9e8902b2e7ae-kube-api-access-xwc2m\" (UniqueName: \"kubernetes.io/projected/e60feb74-e154-496d-88d1-9e8902b2e7ae-kube-api-access-xwc2m\") pod \"e60feb74-e154-496d-88d1-9e8902b2e7ae\" (UID: \"e60feb74-e154-496d-88d1-9e8902b2e7ae\") "
Oct 31 01:22:41.580766 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.574757    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsrpz\" (UniqueName: \"kubernetes.io/projected/f745fa94-dca6-46ac-981e-73ce0831e867-kube-api-access-lsrpz\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:41.580766 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.574783    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f745fa94-dca6-46ac-981e-73ce0831e867-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:41.580766 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.574797    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f745fa94-dca6-46ac-981e-73ce0831e867-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:41.580766 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.574810    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f745fa94-dca6-46ac-981e-73ce0831e867-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:41.580766 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.578671    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e60feb74-e154-496d-88d1-9e8902b2e7ae-plugins-config-volume" pod "e60feb74-e154-496d-88d1-9e8902b2e7ae" (UID: "e60feb74-e154-496d-88d1-9e8902b2e7ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:22:41.580766 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.579602    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdeebb2d-1d8b-49e0-bdee-af282e78e7a1-plugins-config-volume" pod "fdeebb2d-1d8b-49e0-bdee-af282e78e7a1" (UID: "fdeebb2d-1d8b-49e0-bdee-af282e78e7a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:22:41.608049 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.607969    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdeebb2d-1d8b-49e0-bdee-af282e78e7a1-kube-api-access-zj65s" pod "fdeebb2d-1d8b-49e0-bdee-af282e78e7a1" (UID: "fdeebb2d-1d8b-49e0-bdee-af282e78e7a1"). InnerVolumeSpecName "kube-api-access-zj65s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:22:41.611201 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.611152    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60feb74-e154-496d-88d1-9e8902b2e7ae-kube-api-access-xwc2m" pod "e60feb74-e154-496d-88d1-9e8902b2e7ae" (UID: "e60feb74-e154-496d-88d1-9e8902b2e7ae"). InnerVolumeSpecName "kube-api-access-xwc2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:22:41.676188 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.676142    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e60feb74-e154-496d-88d1-9e8902b2e7ae-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:41.676188 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.676191    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdeebb2d-1d8b-49e0-bdee-af282e78e7a1-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:41.676395 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.676207    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj65s\" (UniqueName: \"kubernetes.io/projected/fdeebb2d-1d8b-49e0-bdee-af282e78e7a1-kube-api-access-zj65s\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:41.676395 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.676228    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwc2m\" (UniqueName: \"kubernetes.io/projected/e60feb74-e154-496d-88d1-9e8902b2e7ae-kube-api-access-xwc2m\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:41.701393 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.701358    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"]
Oct 31 01:22:41.713939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.713874    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-pslz7"]
Oct 31 01:22:41.922035 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:41.922004    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f745fa94-dca6-46ac-981e-73ce0831e867" path="/var/lib/kubelet/pods/f745fa94-dca6-46ac-981e-73ce0831e867/volumes"
Oct 31 01:22:42.360705 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:42.360636    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd"
Oct 31 01:22:42.360705 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:42.360654    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd" event={"ID":"e60feb74-e154-496d-88d1-9e8902b2e7ae","Type":"ContainerDied","Data":"8eb86ad0eef6b912a689fc565ead136fd62d47027760345a3fe5dd807a00da56"}
Oct 31 01:22:42.361264 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:42.360746    4327 scope.go:122] "RemoveContainer" containerID="52d3a2d5f329673403c5f94f5fbb577785af0cb07ff7d84b54fab6cb3c29dc84"
Oct 31 01:22:42.362375 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:42.362350    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr" event={"ID":"fdeebb2d-1d8b-49e0-bdee-af282e78e7a1","Type":"ContainerDied","Data":"02c9087a6c492e47b6b52fa851c0ffcef0771c6b11cf976153e70d2959cddaaa"}
Oct 31 01:22:42.362508 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:42.362391    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr"
Oct 31 01:22:42.376102 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:42.376077    4327 scope.go:122] "RemoveContainer" containerID="d1238ead261897c88a71f624d675cec283b8ebd3e15f78a4fee71b3978a69177"
Oct 31 01:22:42.413991 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:42.413953    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd"]
Oct 31 01:22:42.422037 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:42.422004    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-5c9bd48775-ldrrd"]
Oct 31 01:22:42.456300 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:42.456261    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr"]
Oct 31 01:22:42.458868 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:42.458841    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d68696-5d76bd94f8-t4rlr"]
Oct 31 01:22:43.921988 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:43.921946    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e60feb74-e154-496d-88d1-9e8902b2e7ae" path="/var/lib/kubelet/pods/e60feb74-e154-496d-88d1-9e8902b2e7ae/volumes"
Oct 31 01:22:43.922649 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:43.922538    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdeebb2d-1d8b-49e0-bdee-af282e78e7a1" path="/var/lib/kubelet/pods/fdeebb2d-1d8b-49e0-bdee-af282e78e7a1/volumes"
Oct 31 01:22:45.293429 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:22:45.293391    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:22:46.973253 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:46.973217    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc"]
Oct 31 01:22:47.007640 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.007600    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb"]
Oct 31 01:22:47.514034 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.514000    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"]
Oct 31 01:22:47.514736 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.514699    4327 cpu_manager.go:414] "containerMap: removing container" podUID="a811bb1a-42b1-4184-b123-c295547116d2" containerName="istio-proxy"
Oct 31 01:22:47.514736 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.514735    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a811bb1a-42b1-4184-b123-c295547116d2" containerName="istio-proxy"
Oct 31 01:22:47.515033 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.514761    4327 cpu_manager.go:414] "containerMap: removing container" podUID="7549bb85-c79e-4740-b3f0-7f1c793b741a" containerName="payload-processing"
Oct 31 01:22:47.515033 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.514775    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7549bb85-c79e-4740-b3f0-7f1c793b741a" containerName="payload-processing"
Oct 31 01:22:47.515033 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.514818    4327 cpu_manager.go:414] "containerMap: removing container" podUID="b835588e-1042-46bb-8554-243c7dab259d" containerName="payload-pre-processing"
Oct 31 01:22:47.515033 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.514829    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b835588e-1042-46bb-8554-243c7dab259d" containerName="payload-pre-processing"
Oct 31 01:22:47.515033 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.514846    4327 cpu_manager.go:414] "containerMap: removing container" podUID="fdeebb2d-1d8b-49e0-bdee-af282e78e7a1" containerName="payload-processing"
Oct 31 01:22:47.515033 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.514853    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdeebb2d-1d8b-49e0-bdee-af282e78e7a1" containerName="payload-processing"
Oct 31 01:22:47.515033 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.514872    4327 cpu_manager.go:414] "containerMap: removing container" podUID="f745fa94-dca6-46ac-981e-73ce0831e867" containerName="istio-proxy"
Oct 31 01:22:47.515033 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.514880    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f745fa94-dca6-46ac-981e-73ce0831e867" containerName="istio-proxy"
Oct 31 01:22:47.515033 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.514895    4327 cpu_manager.go:414] "containerMap: removing container" podUID="eab68ddb-030b-40ba-a1f9-bac24591237e" containerName="payload-pre-processing"
Oct 31 01:22:47.515033 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.514902    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eab68ddb-030b-40ba-a1f9-bac24591237e" containerName="payload-pre-processing"
Oct 31 01:22:47.515033 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.514915    4327 cpu_manager.go:414] "containerMap: removing container" podUID="e60feb74-e154-496d-88d1-9e8902b2e7ae" containerName="payload-pre-processing"
Oct 31 01:22:47.515033 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.514923    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e60feb74-e154-496d-88d1-9e8902b2e7ae" containerName="payload-pre-processing"
Oct 31 01:22:47.515033 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.514935    4327 cpu_manager.go:414] "containerMap: removing container" podUID="08107827-859a-452d-9d7e-85377ee4b3ab" containerName="main"
Oct 31 01:22:47.515033 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.514942    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08107827-859a-452d-9d7e-85377ee4b3ab" containerName="main"
Oct 31 01:22:47.515783 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.515084    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="a811bb1a-42b1-4184-b123-c295547116d2" containerName="istio-proxy"
Oct 31 01:22:47.515783 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.515108    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="7549bb85-c79e-4740-b3f0-7f1c793b741a" containerName="payload-processing"
Oct 31 01:22:47.515783 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.515131    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="a811bb1a-42b1-4184-b123-c295547116d2" containerName="istio-proxy"
Oct 31 01:22:47.515783 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.515145    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="b835588e-1042-46bb-8554-243c7dab259d" containerName="payload-pre-processing"
Oct 31 01:22:47.515783 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.515163    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdeebb2d-1d8b-49e0-bdee-af282e78e7a1" containerName="payload-processing"
Oct 31 01:22:47.515783 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.515181    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="f745fa94-dca6-46ac-981e-73ce0831e867" containerName="istio-proxy"
Oct 31 01:22:47.515783 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.515201    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="a24c7c0a-782b-4d04-976d-9f62a964250d" containerName="istio-proxy"
Oct 31 01:22:47.515783 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.515217    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="eab68ddb-030b-40ba-a1f9-bac24591237e" containerName="payload-pre-processing"
Oct 31 01:22:47.515783 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.515233    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="e60feb74-e154-496d-88d1-9e8902b2e7ae" containerName="payload-pre-processing"
Oct 31 01:22:47.515783 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.515243    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="08107827-859a-452d-9d7e-85377ee4b3ab" containerName="main"
Oct 31 01:22:47.520977 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.520951    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.540104 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.540062    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"]
Oct 31 01:22:47.550450 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.550394    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.550615 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.550532    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.550615 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.550567    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.550729 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.550616    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.550729 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.550666    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.550815 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.550768    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.550815 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.550804    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.550888 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.550842    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9vfd\" (UniqueName: \"kubernetes.io/projected/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-kube-api-access-l9vfd\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.550965 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.550949    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.652732 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.652695    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.652922 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.652751    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9vfd\" (UniqueName: \"kubernetes.io/projected/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-kube-api-access-l9vfd\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.652922 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.652798    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.652922 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.652839    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.653113 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.652922    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.653113 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.652947    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.653113 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.652994    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.653113 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.653040    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.653113 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.653098    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.653370 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.653277    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.653416 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.653374    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.653466 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.653430    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.653505 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.653456    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.653902 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.653878    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.655289 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.655269    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.655419 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.655400    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.676216 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.676187    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.676363 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.676339    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9vfd\" (UniqueName: \"kubernetes.io/projected/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-kube-api-access-l9vfd\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:47.839593 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:47.839512    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:48.151987 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:22:48.151956    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f5a2fa_d0dd_43f4_ace4_e02a8aa4197f.slice/crio-b5d6bc6f6e1897e0ef7bfc8c1e7aa3f7e2243d57a70ce4e49c7040a83c5e0986 WatchSource:0}: Error finding container b5d6bc6f6e1897e0ef7bfc8c1e7aa3f7e2243d57a70ce4e49c7040a83c5e0986: Status 404 returned error can't find the container with id b5d6bc6f6e1897e0ef7bfc8c1e7aa3f7e2243d57a70ce4e49c7040a83c5e0986
Oct 31 01:22:48.152314 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:48.152058    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"]
Oct 31 01:22:48.154449 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:48.154412    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:22:48.154574 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:48.154490    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:22:48.154574 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:48.154538    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:22:48.400893 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:48.400580    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5" event={"ID":"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f","Type":"ContainerStarted","Data":"8757d9ce7e27cc54551cc354516e10fbba4b1e5f2e0b1cf13f4b514911f9d2d0"}
Oct 31 01:22:48.400893 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:48.400646    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5" event={"ID":"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f","Type":"ContainerStarted","Data":"b5d6bc6f6e1897e0ef7bfc8c1e7aa3f7e2243d57a70ce4e49c7040a83c5e0986"}
Oct 31 01:22:48.449182 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:48.449124    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5" podStartSLOduration=1.449109071 podStartE2EDuration="1.449109071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:48.447606424 +0000 UTC m=+3828.669641968" watchObservedRunningTime="2026-10-31 01:22:48.449109071 +0000 UTC m=+3828.671144613"
Oct 31 01:22:48.841593 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:48.840389    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:49.842429 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:49.842395    4327 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 31 01:22:49.842962 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:49.842449    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5" podUID="41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 31 01:22:50.842158 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:50.842115    4327 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 31 01:22:50.842448 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:50.842420    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5" podUID="41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 31 01:22:51.841760 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:51.841111    4327 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 31 01:22:51.841760 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:51.841173    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5" podUID="41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 31 01:22:51.973635 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:51.973595    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc" podUID="0d05d663-30be-4ee3-97c8-801ab8d0f885" containerName="payload-processing" containerID="cri-o://2e284c3bc5b4cfb9abf67cd2588e747e26e5d0406d1c088f8dc926484dc13127" gracePeriod=25
Oct 31 01:22:52.008077 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:52.008039    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb" podUID="2c725e03-2ab7-49a9-8251-606dc8f70d4f" containerName="payload-pre-processing" containerID="cri-o://451ebdddadb823ba6ff721b3971825b3fa8263929be2fd584f61216bb89a239d" gracePeriod=25
Oct 31 01:22:52.843719 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:52.843565    4327 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 31 01:22:52.843719 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:52.843639    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5" podUID="41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 31 01:22:53.841706 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:53.840771    4327 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 31 01:22:53.841706 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:53.840828    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5" podUID="41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 31 01:22:54.844827 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:54.844794    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:54.845277 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:54.845257    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:54.846088 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:54.846070    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:22:57.058259 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:57.058211    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb" podUID="2c725e03-2ab7-49a9-8251-606dc8f70d4f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:22:57.058767 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:57.058270    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc" podUID="0d05d663-30be-4ee3-97c8-801ab8d0f885" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:22:57.590805 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:57.590767    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg"]
Oct 31 01:22:57.591264 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:57.591232    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg" podUID="fac0cd38-95b3-4875-b81f-3e04f000f4b3" containerName="maas-api" containerID="cri-o://da2739efeca4b39db30e2172bcc2fae27a02936e1bcbb4f13d6926a1307e1ab7" gracePeriod=30
Oct 31 01:22:57.875710 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:57.875462    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg"
Oct 31 01:22:58.020270 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:58.020229    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fac0cd38-95b3-4875-b81f-3e04f000f4b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fac0cd38-95b3-4875-b81f-3e04f000f4b3-metrics-certs\") pod \"fac0cd38-95b3-4875-b81f-3e04f000f4b3\" (UID: \"fac0cd38-95b3-4875-b81f-3e04f000f4b3\") "
Oct 31 01:22:58.020482 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:58.020401    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fac0cd38-95b3-4875-b81f-3e04f000f4b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fac0cd38-95b3-4875-b81f-3e04f000f4b3-openshift-service-ca\") pod \"fac0cd38-95b3-4875-b81f-3e04f000f4b3\" (UID: \"fac0cd38-95b3-4875-b81f-3e04f000f4b3\") "
Oct 31 01:22:58.020482 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:58.020453    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fac0cd38-95b3-4875-b81f-3e04f000f4b3-kube-api-access-5q4xm\" (UniqueName: \"kubernetes.io/projected/fac0cd38-95b3-4875-b81f-3e04f000f4b3-kube-api-access-5q4xm\") pod \"fac0cd38-95b3-4875-b81f-3e04f000f4b3\" (UID: \"fac0cd38-95b3-4875-b81f-3e04f000f4b3\") "
Oct 31 01:22:58.020618 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:58.020535    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fac0cd38-95b3-4875-b81f-3e04f000f4b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fac0cd38-95b3-4875-b81f-3e04f000f4b3-maas-api-tls\") pod \"fac0cd38-95b3-4875-b81f-3e04f000f4b3\" (UID: \"fac0cd38-95b3-4875-b81f-3e04f000f4b3\") "
Oct 31 01:22:58.020862 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:58.020823    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fac0cd38-95b3-4875-b81f-3e04f000f4b3-openshift-service-ca" pod "fac0cd38-95b3-4875-b81f-3e04f000f4b3" (UID: "fac0cd38-95b3-4875-b81f-3e04f000f4b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:22:58.021110 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:58.021080    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fac0cd38-95b3-4875-b81f-3e04f000f4b3-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:58.022648 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:58.022618    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fac0cd38-95b3-4875-b81f-3e04f000f4b3-metrics-certs" pod "fac0cd38-95b3-4875-b81f-3e04f000f4b3" (UID: "fac0cd38-95b3-4875-b81f-3e04f000f4b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:22:58.022998 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:58.022980    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fac0cd38-95b3-4875-b81f-3e04f000f4b3-maas-api-tls" pod "fac0cd38-95b3-4875-b81f-3e04f000f4b3" (UID: "fac0cd38-95b3-4875-b81f-3e04f000f4b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:22:58.023264 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:58.023250    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac0cd38-95b3-4875-b81f-3e04f000f4b3-kube-api-access-5q4xm" pod "fac0cd38-95b3-4875-b81f-3e04f000f4b3" (UID: "fac0cd38-95b3-4875-b81f-3e04f000f4b3"). InnerVolumeSpecName "kube-api-access-5q4xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:22:58.122357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:58.122324    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q4xm\" (UniqueName: \"kubernetes.io/projected/fac0cd38-95b3-4875-b81f-3e04f000f4b3-kube-api-access-5q4xm\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:58.122357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:58.122355    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fac0cd38-95b3-4875-b81f-3e04f000f4b3-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:58.122805 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:58.122368    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fac0cd38-95b3-4875-b81f-3e04f000f4b3-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:22:58.523256 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:58.523215    4327 generic.go:356] "Generic (PLEG): container finished" podID="fac0cd38-95b3-4875-b81f-3e04f000f4b3" containerID="da2739efeca4b39db30e2172bcc2fae27a02936e1bcbb4f13d6926a1307e1ab7" exitCode=0
Oct 31 01:22:58.523411 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:58.523262    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg"
Oct 31 01:22:58.523411 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:58.523266    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg" event={"ID":"fac0cd38-95b3-4875-b81f-3e04f000f4b3","Type":"ContainerDied","Data":"da2739efeca4b39db30e2172bcc2fae27a02936e1bcbb4f13d6926a1307e1ab7"}
Oct 31 01:22:58.523411 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:58.523388    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg" event={"ID":"fac0cd38-95b3-4875-b81f-3e04f000f4b3","Type":"ContainerDied","Data":"1656a38b920a021538a911880ef87ea7611035ffc3af5e9c4a74e8afd00c6873"}
Oct 31 01:22:58.523604 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:58.523414    4327 scope.go:122] "RemoveContainer" containerID="da2739efeca4b39db30e2172bcc2fae27a02936e1bcbb4f13d6926a1307e1ab7"
Oct 31 01:22:58.534170 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:58.534147    4327 scope.go:122] "RemoveContainer" containerID="da2739efeca4b39db30e2172bcc2fae27a02936e1bcbb4f13d6926a1307e1ab7"
Oct 31 01:22:58.534475 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:22:58.534454    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da2739efeca4b39db30e2172bcc2fae27a02936e1bcbb4f13d6926a1307e1ab7\": container with ID starting with da2739efeca4b39db30e2172bcc2fae27a02936e1bcbb4f13d6926a1307e1ab7 not found: ID does not exist" containerID="da2739efeca4b39db30e2172bcc2fae27a02936e1bcbb4f13d6926a1307e1ab7"
Oct 31 01:22:58.534543 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:58.534483    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da2739efeca4b39db30e2172bcc2fae27a02936e1bcbb4f13d6926a1307e1ab7"} err="failed to get container status \"da2739efeca4b39db30e2172bcc2fae27a02936e1bcbb4f13d6926a1307e1ab7\": rpc error: code = NotFound desc = could not find container \"da2739efeca4b39db30e2172bcc2fae27a02936e1bcbb4f13d6926a1307e1ab7\": container with ID starting with da2739efeca4b39db30e2172bcc2fae27a02936e1bcbb4f13d6926a1307e1ab7 not found: ID does not exist"
Oct 31 01:22:58.584665 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:58.584629    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg"]
Oct 31 01:22:58.608833 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:58.608795    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-be301b-c6fcc7f5d-qshqg"]
Oct 31 01:22:59.920576 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:22:59.920542    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fac0cd38-95b3-4875-b81f-3e04f000f4b3" path="/var/lib/kubelet/pods/fac0cd38-95b3-4875-b81f-3e04f000f4b3/volumes"
Oct 31 01:23:00.616248 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:00.616213    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr"]
Oct 31 01:23:00.673478 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:00.673440    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl"]
Oct 31 01:23:02.519704 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.519649    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"]
Oct 31 01:23:02.520311 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.520294    4327 cpu_manager.go:414] "containerMap: removing container" podUID="fac0cd38-95b3-4875-b81f-3e04f000f4b3" containerName="maas-api"
Oct 31 01:23:02.520359 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.520312    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fac0cd38-95b3-4875-b81f-3e04f000f4b3" containerName="maas-api"
Oct 31 01:23:02.520359 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.520347    4327 cpu_manager.go:414] "containerMap: removing container" podUID="a811bb1a-42b1-4184-b123-c295547116d2" containerName="istio-proxy"
Oct 31 01:23:02.520359 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.520353    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a811bb1a-42b1-4184-b123-c295547116d2" containerName="istio-proxy"
Oct 31 01:23:02.520466 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.520446    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="fac0cd38-95b3-4875-b81f-3e04f000f4b3" containerName="maas-api"
Oct 31 01:23:02.525211 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.525190    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.551954 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.551923    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"]
Oct 31 01:23:02.688568 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.688528    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.688800 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.688574    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.688800 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.688643    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.688800 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.688716    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttvg9\" (UniqueName: \"kubernetes.io/projected/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-kube-api-access-ttvg9\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.688939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.688815    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.688939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.688863    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.688939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.688916    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.689049 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.689027    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.689095 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.689081    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.790648 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.790553    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.790648 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.790617    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.791011 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.790710    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.791011 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.790819    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.791011 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.790883    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.791011 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.790968    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.791248 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.791016    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.791248 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.791115    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.791248 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.791171    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.791465 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.791248    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.791465 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.791314    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.791465 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.791351    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.791465 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.791356    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttvg9\" (UniqueName: \"kubernetes.io/projected/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-kube-api-access-ttvg9\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.791637 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.791484    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.793027 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.793007    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.793252 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.793236    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.819393 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.819364    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttvg9\" (UniqueName: \"kubernetes.io/projected/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-kube-api-access-ttvg9\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.821870 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.821844    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:02.840870 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:02.840838    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:03.159215 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:23:03.159183    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4e63bc_59e0_4eeb_85ba_fdc28515aeff.slice/crio-3fea9f0a591b4a15ed35c5e266e2d8f1be4851008c7cfe3ffb2d5d4a13a2fe1d WatchSource:0}: Error finding container 3fea9f0a591b4a15ed35c5e266e2d8f1be4851008c7cfe3ffb2d5d4a13a2fe1d: Status 404 returned error can't find the container with id 3fea9f0a591b4a15ed35c5e266e2d8f1be4851008c7cfe3ffb2d5d4a13a2fe1d
Oct 31 01:23:03.159873 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:03.159759    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"]
Oct 31 01:23:03.161782 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:03.161753    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:23:03.161866 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:03.161815    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:23:03.161866 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:03.161843    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:23:03.556898 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:03.556799    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm" event={"ID":"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff","Type":"ContainerStarted","Data":"76c6b1d4f8dd44035e2ff9670a893e481bb03bce08718173b55149d500df7b3a"}
Oct 31 01:23:03.556898 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:03.556867    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm" event={"ID":"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff","Type":"ContainerStarted","Data":"3fea9f0a591b4a15ed35c5e266e2d8f1be4851008c7cfe3ffb2d5d4a13a2fe1d"}
Oct 31 01:23:03.605511 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:03.605454    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm" podStartSLOduration=1.605441011 podStartE2EDuration="1.605441011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:03.602191782 +0000 UTC m=+3843.824227323" watchObservedRunningTime="2026-10-31 01:23:03.605441011 +0000 UTC m=+3843.827476554"
Oct 31 01:23:03.841466 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:03.841387    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:04.289778 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:04.289736    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"]
Oct 31 01:23:04.289995 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:04.289954    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j" podUID="2387e9d0-6540-45b2-b624-78f6f63e26f5" containerName="istio-proxy" containerID="cri-o://04cd907891d7c24397e948a3ec04a5c4bac026a2203197a7909ef12f32aad448" gracePeriod=30
Oct 31 01:23:04.843792 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:04.843666    4327 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Oct 31 01:23:04.843792 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:04.843756    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm" podUID="5c4e63bc-59e0-4eeb-85ba-fdc28515aeff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Oct 31 01:23:05.620810 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:05.620758    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr" podUID="c853f186-bade-4124-b526-438799578a2c" containerName="payload-processing" containerID="cri-o://042d70cc4386f58baf06a5b6222cc988dc62e9693ed0a984a19734153f786939" gracePeriod=25
Oct 31 01:23:05.674309 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:05.674204    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl" podUID="9dcfe474-6b38-4ada-9272-a48f011722b3" containerName="payload-pre-processing" containerID="cri-o://c67c8fa82494def93542d40cbe743e1306a074e8370d63320f3ece0a5178bfac" gracePeriod=25
Oct 31 01:23:05.840128 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:23:05.840067    4327 logging.go:55] [core] [Channel #15332 SubChannel #15333]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.185:50052", ServerName: "10.133.0.185:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.185:50052: connect: connection refused"
Oct 31 01:23:05.844746 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:05.844269    4327 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Oct 31 01:23:05.844746 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:05.844326    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm" podUID="5c4e63bc-59e0-4eeb-85ba-fdc28515aeff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Oct 31 01:23:05.985739 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:05.983443    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr"
Oct 31 01:23:06.060734 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.059077    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c853f186-bade-4124-b526-438799578a2c-kube-api-access-kt2xd\" (UniqueName: \"kubernetes.io/projected/c853f186-bade-4124-b526-438799578a2c-kube-api-access-kt2xd\") pod \"c853f186-bade-4124-b526-438799578a2c\" (UID: \"c853f186-bade-4124-b526-438799578a2c\") "
Oct 31 01:23:06.060734 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.059468    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c853f186-bade-4124-b526-438799578a2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c853f186-bade-4124-b526-438799578a2c-plugins-config-volume\") pod \"c853f186-bade-4124-b526-438799578a2c\" (UID: \"c853f186-bade-4124-b526-438799578a2c\") "
Oct 31 01:23:06.060734 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.059944    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c853f186-bade-4124-b526-438799578a2c-plugins-config-volume" pod "c853f186-bade-4124-b526-438799578a2c" (UID: "c853f186-bade-4124-b526-438799578a2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:06.060734 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.060563    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c853f186-bade-4124-b526-438799578a2c-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:06.062049 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.062016    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c853f186-bade-4124-b526-438799578a2c-kube-api-access-kt2xd" pod "c853f186-bade-4124-b526-438799578a2c" (UID: "c853f186-bade-4124-b526-438799578a2c"). InnerVolumeSpecName "kube-api-access-kt2xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:06.066062 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.066037    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl"
Oct 31 01:23:06.161390 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.161301    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dcfe474-6b38-4ada-9272-a48f011722b3-kube-api-access-wg5wv\" (UniqueName: \"kubernetes.io/projected/9dcfe474-6b38-4ada-9272-a48f011722b3-kube-api-access-wg5wv\") pod \"9dcfe474-6b38-4ada-9272-a48f011722b3\" (UID: \"9dcfe474-6b38-4ada-9272-a48f011722b3\") "
Oct 31 01:23:06.161580 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.161446    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dcfe474-6b38-4ada-9272-a48f011722b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dcfe474-6b38-4ada-9272-a48f011722b3-plugins-config-volume\") pod \"9dcfe474-6b38-4ada-9272-a48f011722b3\" (UID: \"9dcfe474-6b38-4ada-9272-a48f011722b3\") "
Oct 31 01:23:06.161882 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.161862    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt2xd\" (UniqueName: \"kubernetes.io/projected/c853f186-bade-4124-b526-438799578a2c-kube-api-access-kt2xd\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:06.161971 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.161952    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dcfe474-6b38-4ada-9272-a48f011722b3-plugins-config-volume" pod "9dcfe474-6b38-4ada-9272-a48f011722b3" (UID: "9dcfe474-6b38-4ada-9272-a48f011722b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:06.164059 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.164037    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dcfe474-6b38-4ada-9272-a48f011722b3-kube-api-access-wg5wv" pod "9dcfe474-6b38-4ada-9272-a48f011722b3" (UID: "9dcfe474-6b38-4ada-9272-a48f011722b3"). InnerVolumeSpecName "kube-api-access-wg5wv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:06.263333 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.263297    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg5wv\" (UniqueName: \"kubernetes.io/projected/9dcfe474-6b38-4ada-9272-a48f011722b3-kube-api-access-wg5wv\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:06.263333 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.263334    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dcfe474-6b38-4ada-9272-a48f011722b3-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:06.576058 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.575956    4327 generic.go:356] "Generic (PLEG): container finished" podID="9dcfe474-6b38-4ada-9272-a48f011722b3" containerID="c67c8fa82494def93542d40cbe743e1306a074e8370d63320f3ece0a5178bfac" exitCode=0
Oct 31 01:23:06.576058 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.576010    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl"
Oct 31 01:23:06.576058 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.576010    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl" event={"ID":"9dcfe474-6b38-4ada-9272-a48f011722b3","Type":"ContainerDied","Data":"c67c8fa82494def93542d40cbe743e1306a074e8370d63320f3ece0a5178bfac"}
Oct 31 01:23:06.576347 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.576073    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl" event={"ID":"9dcfe474-6b38-4ada-9272-a48f011722b3","Type":"ContainerDied","Data":"2eeaa57605f99059c9b46695744e7ea2e93620861c508aa49a55a57c5824162d"}
Oct 31 01:23:06.576347 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.576106    4327 scope.go:122] "RemoveContainer" containerID="c67c8fa82494def93542d40cbe743e1306a074e8370d63320f3ece0a5178bfac"
Oct 31 01:23:06.577811 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.577782    4327 generic.go:356] "Generic (PLEG): container finished" podID="c853f186-bade-4124-b526-438799578a2c" containerID="042d70cc4386f58baf06a5b6222cc988dc62e9693ed0a984a19734153f786939" exitCode=0
Oct 31 01:23:06.577967 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.577882    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr"
Oct 31 01:23:06.577967 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.577924    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr" event={"ID":"c853f186-bade-4124-b526-438799578a2c","Type":"ContainerDied","Data":"042d70cc4386f58baf06a5b6222cc988dc62e9693ed0a984a19734153f786939"}
Oct 31 01:23:06.578101 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.577970    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr" event={"ID":"c853f186-bade-4124-b526-438799578a2c","Type":"ContainerDied","Data":"806b4916e74ae44917e6f3481938ae00ee58e31760a2a0506b594447a58f5dc4"}
Oct 31 01:23:06.587554 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.587536    4327 scope.go:122] "RemoveContainer" containerID="c67c8fa82494def93542d40cbe743e1306a074e8370d63320f3ece0a5178bfac"
Oct 31 01:23:06.587884 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:23:06.587853    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c67c8fa82494def93542d40cbe743e1306a074e8370d63320f3ece0a5178bfac\": container with ID starting with c67c8fa82494def93542d40cbe743e1306a074e8370d63320f3ece0a5178bfac not found: ID does not exist" containerID="c67c8fa82494def93542d40cbe743e1306a074e8370d63320f3ece0a5178bfac"
Oct 31 01:23:06.587994 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.587884    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c67c8fa82494def93542d40cbe743e1306a074e8370d63320f3ece0a5178bfac"} err="failed to get container status \"c67c8fa82494def93542d40cbe743e1306a074e8370d63320f3ece0a5178bfac\": rpc error: code = NotFound desc = could not find container \"c67c8fa82494def93542d40cbe743e1306a074e8370d63320f3ece0a5178bfac\": container with ID starting with c67c8fa82494def93542d40cbe743e1306a074e8370d63320f3ece0a5178bfac not found: ID does not exist"
Oct 31 01:23:06.587994 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.587900    4327 scope.go:122] "RemoveContainer" containerID="042d70cc4386f58baf06a5b6222cc988dc62e9693ed0a984a19734153f786939"
Oct 31 01:23:06.683701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.679439    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr"]
Oct 31 01:23:06.693179 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.686656    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr"]
Oct 31 01:23:06.693179 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.686923    4327 scope.go:122] "RemoveContainer" containerID="042d70cc4386f58baf06a5b6222cc988dc62e9693ed0a984a19734153f786939"
Oct 31 01:23:06.693179 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:23:06.687471    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042d70cc4386f58baf06a5b6222cc988dc62e9693ed0a984a19734153f786939\": container with ID starting with 042d70cc4386f58baf06a5b6222cc988dc62e9693ed0a984a19734153f786939 not found: ID does not exist" containerID="042d70cc4386f58baf06a5b6222cc988dc62e9693ed0a984a19734153f786939"
Oct 31 01:23:06.693179 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.687512    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042d70cc4386f58baf06a5b6222cc988dc62e9693ed0a984a19734153f786939"} err="failed to get container status \"042d70cc4386f58baf06a5b6222cc988dc62e9693ed0a984a19734153f786939\": rpc error: code = NotFound desc = could not find container \"042d70cc4386f58baf06a5b6222cc988dc62e9693ed0a984a19734153f786939\": container with ID starting with 042d70cc4386f58baf06a5b6222cc988dc62e9693ed0a984a19734153f786939 not found: ID does not exist"
Oct 31 01:23:06.741704 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.741311    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl"]
Oct 31 01:23:06.741704 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.741382    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99a5c2-b794bc94c-nvpzl"]
Oct 31 01:23:06.840757 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.839876    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99a5c2-7654644d6f-8ndnr" podUID="c853f186-bade-4124-b526-438799578a2c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.185:50052\" within 1s: context deadline exceeded"
Oct 31 01:23:06.842271 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.842149    4327 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Oct 31 01:23:06.842271 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:06.842198    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm" podUID="5c4e63bc-59e0-4eeb-85ba-fdc28515aeff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Oct 31 01:23:07.057732 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:07.057672    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc" podUID="0d05d663-30be-4ee3-97c8-801ab8d0f885" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:23:07.058154 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:07.057745    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb" podUID="2c725e03-2ab7-49a9-8251-606dc8f70d4f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:23:07.842183 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:07.842145    4327 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Oct 31 01:23:07.842416 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:07.842209    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm" podUID="5c4e63bc-59e0-4eeb-85ba-fdc28515aeff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Oct 31 01:23:07.944341 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:07.944305    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dcfe474-6b38-4ada-9272-a48f011722b3" path="/var/lib/kubelet/pods/9dcfe474-6b38-4ada-9272-a48f011722b3/volumes"
Oct 31 01:23:07.944760 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:07.944747    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c853f186-bade-4124-b526-438799578a2c" path="/var/lib/kubelet/pods/c853f186-bade-4124-b526-438799578a2c/volumes"
Oct 31 01:23:08.672000 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:08.671954    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:23:08.672419 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:08.672025    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j" podUID="2387e9d0-6540-45b2-b624-78f6f63e26f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:23:08.842151 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:08.842103    4327 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Oct 31 01:23:08.842389 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:08.842179    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm" podUID="5c4e63bc-59e0-4eeb-85ba-fdc28515aeff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Oct 31 01:23:09.439619 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.438900    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n"]
Oct 31 01:23:09.455253 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.451058    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n" podUID="ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf" containerName="maas-api" containerID="cri-o://64d1de9a8acedbd1fdbedc370885c798ae7ee7fdcab3f106816cfbebc11b52f2" gracePeriod=30
Oct 31 01:23:09.602510 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.602452    4327 generic.go:356] "Generic (PLEG): container finished" podID="ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf" containerID="64d1de9a8acedbd1fdbedc370885c798ae7ee7fdcab3f106816cfbebc11b52f2" exitCode=0
Oct 31 01:23:09.602657 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.602480    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n" event={"ID":"ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf","Type":"ContainerDied","Data":"64d1de9a8acedbd1fdbedc370885c798ae7ee7fdcab3f106816cfbebc11b52f2"}
Oct 31 01:23:09.604939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.604912    4327 generic.go:356] "Generic (PLEG): container finished" podID="2387e9d0-6540-45b2-b624-78f6f63e26f5" containerID="04cd907891d7c24397e948a3ec04a5c4bac026a2203197a7909ef12f32aad448" exitCode=0
Oct 31 01:23:09.605052 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.604979    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j" event={"ID":"2387e9d0-6540-45b2-b624-78f6f63e26f5","Type":"ContainerDied","Data":"04cd907891d7c24397e948a3ec04a5c4bac026a2203197a7909ef12f32aad448"}
Oct 31 01:23:09.615588 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.615551    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g"]
Oct 31 01:23:09.615897 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.615849    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g" podUID="ceab7625-0443-4896-a735-5e917167771a" containerName="payload-processing" containerID="cri-o://624c37fbb148f3926a087d15392e4b6e3f0b14c956f58b1916f9c093dce70d8d" gracePeriod=30
Oct 31 01:23:09.633933 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.633897    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k"]
Oct 31 01:23:09.634391 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.634356    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k" podUID="edf4f19d-74fd-4f83-96d2-6c66d711fa24" containerName="payload-pre-processing" containerID="cri-o://54ca17c793ec754eb9d63059cf6fc1879105de9643defc7d2b65fbdd5616818a" gracePeriod=30
Oct 31 01:23:09.748382 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.748342    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:23:09.785572 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.785541    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n"
Oct 31 01:23:09.821500 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.821453    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-workload-socket\") pod \"2387e9d0-6540-45b2-b624-78f6f63e26f5\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") "
Oct 31 01:23:09.821654 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.821538    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-openshift-service-ca\") pod \"ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf\" (UID: \"ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf\") "
Oct 31 01:23:09.821654 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.821632    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-workload-certs\") pod \"2387e9d0-6540-45b2-b624-78f6f63e26f5\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") "
Oct 31 01:23:09.821793 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.821753    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-podinfo\") pod \"2387e9d0-6540-45b2-b624-78f6f63e26f5\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") "
Oct 31 01:23:09.821853 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.821838    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-envoy\") pod \"2387e9d0-6540-45b2-b624-78f6f63e26f5\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") "
Oct 31 01:23:09.821896 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.821869    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2387e9d0-6540-45b2-b624-78f6f63e26f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2387e9d0-6540-45b2-b624-78f6f63e26f5-istiod-ca-cert\") pod \"2387e9d0-6540-45b2-b624-78f6f63e26f5\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") "
Oct 31 01:23:09.821949 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.821932    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-token\" (UniqueName: \"kubernetes.io/projected/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-token\") pod \"2387e9d0-6540-45b2-b624-78f6f63e26f5\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") "
Oct 31 01:23:09.821988 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.821969    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2387e9d0-6540-45b2-b624-78f6f63e26f5-kube-api-access-4nbh2\" (UniqueName: \"kubernetes.io/projected/2387e9d0-6540-45b2-b624-78f6f63e26f5-kube-api-access-4nbh2\") pod \"2387e9d0-6540-45b2-b624-78f6f63e26f5\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") "
Oct 31 01:23:09.822063 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.822001    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-maas-api-tls\") pod \"ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf\" (UID: \"ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf\") "
Oct 31 01:23:09.822063 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.822040    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-kube-api-access-gscks\" (UniqueName: \"kubernetes.io/projected/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-kube-api-access-gscks\") pod \"ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf\" (UID: \"ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf\") "
Oct 31 01:23:09.822141 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.822065    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-credential-socket\") pod \"2387e9d0-6540-45b2-b624-78f6f63e26f5\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") "
Oct 31 01:23:09.822141 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.822094    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-metrics-certs\") pod \"ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf\" (UID: \"ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf\") "
Oct 31 01:23:09.822141 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.822126    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-data\") pod \"2387e9d0-6540-45b2-b624-78f6f63e26f5\" (UID: \"2387e9d0-6540-45b2-b624-78f6f63e26f5\") "
Oct 31 01:23:09.823916 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.822869    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-data" pod "2387e9d0-6540-45b2-b624-78f6f63e26f5" (UID: "2387e9d0-6540-45b2-b624-78f6f63e26f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:09.823916 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.822993    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-workload-certs" pod "2387e9d0-6540-45b2-b624-78f6f63e26f5" (UID: "2387e9d0-6540-45b2-b624-78f6f63e26f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:09.823916 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.823191    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2387e9d0-6540-45b2-b624-78f6f63e26f5-istiod-ca-cert" pod "2387e9d0-6540-45b2-b624-78f6f63e26f5" (UID: "2387e9d0-6540-45b2-b624-78f6f63e26f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:09.823916 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.823318    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-openshift-service-ca" pod "ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf" (UID: "ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:09.823916 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.823414    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-workload-socket" pod "2387e9d0-6540-45b2-b624-78f6f63e26f5" (UID: "2387e9d0-6540-45b2-b624-78f6f63e26f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:09.823916 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.823627    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-credential-socket" pod "2387e9d0-6540-45b2-b624-78f6f63e26f5" (UID: "2387e9d0-6540-45b2-b624-78f6f63e26f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:09.831718 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.829943    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-metrics-certs" pod "ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf" (UID: "ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:23:09.831718 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.830068    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-podinfo" pod "2387e9d0-6540-45b2-b624-78f6f63e26f5" (UID: "2387e9d0-6540-45b2-b624-78f6f63e26f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:23:09.831718 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.830080    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2387e9d0-6540-45b2-b624-78f6f63e26f5-kube-api-access-4nbh2" pod "2387e9d0-6540-45b2-b624-78f6f63e26f5" (UID: "2387e9d0-6540-45b2-b624-78f6f63e26f5"). InnerVolumeSpecName "kube-api-access-4nbh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:09.831718 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.830208    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-envoy" pod "2387e9d0-6540-45b2-b624-78f6f63e26f5" (UID: "2387e9d0-6540-45b2-b624-78f6f63e26f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:09.831718 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.830314    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-token" pod "2387e9d0-6540-45b2-b624-78f6f63e26f5" (UID: "2387e9d0-6540-45b2-b624-78f6f63e26f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:09.831718 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.831534    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-kube-api-access-gscks" pod "ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf" (UID: "ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf"). InnerVolumeSpecName "kube-api-access-gscks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:09.832156 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.831742    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-maas-api-tls" pod "ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf" (UID: "ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:23:09.848938 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.848895    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:09.852536 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.850146    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:09.852536 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.851864    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:09.924971 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.924703    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:09.924971 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.924735    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nbh2\" (UniqueName: \"kubernetes.io/projected/2387e9d0-6540-45b2-b624-78f6f63e26f5-kube-api-access-4nbh2\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:09.924971 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.924749    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:09.924971 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.924766    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gscks\" (UniqueName: \"kubernetes.io/projected/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-kube-api-access-gscks\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:09.924971 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.924781    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:09.924971 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.924794    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:09.924971 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.924807    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:09.924971 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.924820    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:09.924971 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.924831    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:09.924971 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.924843    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:09.924971 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.924856    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:09.924971 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.924866    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2387e9d0-6540-45b2-b624-78f6f63e26f5-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:09.924971 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:09.924879    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2387e9d0-6540-45b2-b624-78f6f63e26f5-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:10.010717 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.003058    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g"
Oct 31 01:23:10.037283 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.037245    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceab7625-0443-4896-a735-5e917167771a-kube-api-access-9flcx\" (UniqueName: \"kubernetes.io/projected/ceab7625-0443-4896-a735-5e917167771a-kube-api-access-9flcx\") pod \"ceab7625-0443-4896-a735-5e917167771a\" (UID: \"ceab7625-0443-4896-a735-5e917167771a\") "
Oct 31 01:23:10.037459 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.037357    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceab7625-0443-4896-a735-5e917167771a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceab7625-0443-4896-a735-5e917167771a-plugins-config-volume\") pod \"ceab7625-0443-4896-a735-5e917167771a\" (UID: \"ceab7625-0443-4896-a735-5e917167771a\") "
Oct 31 01:23:10.055275 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.055113    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceab7625-0443-4896-a735-5e917167771a-plugins-config-volume" pod "ceab7625-0443-4896-a735-5e917167771a" (UID: "ceab7625-0443-4896-a735-5e917167771a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:10.096914 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.095342    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceab7625-0443-4896-a735-5e917167771a-kube-api-access-9flcx" pod "ceab7625-0443-4896-a735-5e917167771a" (UID: "ceab7625-0443-4896-a735-5e917167771a"). InnerVolumeSpecName "kube-api-access-9flcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:10.106758 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.106724    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k"
Oct 31 01:23:10.139188 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.139149    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edf4f19d-74fd-4f83-96d2-6c66d711fa24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edf4f19d-74fd-4f83-96d2-6c66d711fa24-plugins-config-volume\") pod \"edf4f19d-74fd-4f83-96d2-6c66d711fa24\" (UID: \"edf4f19d-74fd-4f83-96d2-6c66d711fa24\") "
Oct 31 01:23:10.139383 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.139262    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edf4f19d-74fd-4f83-96d2-6c66d711fa24-kube-api-access-jp4wd\" (UniqueName: \"kubernetes.io/projected/edf4f19d-74fd-4f83-96d2-6c66d711fa24-kube-api-access-jp4wd\") pod \"edf4f19d-74fd-4f83-96d2-6c66d711fa24\" (UID: \"edf4f19d-74fd-4f83-96d2-6c66d711fa24\") "
Oct 31 01:23:10.139610 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.139580    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edf4f19d-74fd-4f83-96d2-6c66d711fa24-plugins-config-volume" pod "edf4f19d-74fd-4f83-96d2-6c66d711fa24" (UID: "edf4f19d-74fd-4f83-96d2-6c66d711fa24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:10.139851 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.139826    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9flcx\" (UniqueName: \"kubernetes.io/projected/ceab7625-0443-4896-a735-5e917167771a-kube-api-access-9flcx\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:10.139911 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.139853    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edf4f19d-74fd-4f83-96d2-6c66d711fa24-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:10.139911 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.139863    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceab7625-0443-4896-a735-5e917167771a-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:10.141381 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.141359    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf4f19d-74fd-4f83-96d2-6c66d711fa24-kube-api-access-jp4wd" pod "edf4f19d-74fd-4f83-96d2-6c66d711fa24" (UID: "edf4f19d-74fd-4f83-96d2-6c66d711fa24"). InnerVolumeSpecName "kube-api-access-jp4wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:10.241521 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.241471    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp4wd\" (UniqueName: \"kubernetes.io/projected/edf4f19d-74fd-4f83-96d2-6c66d711fa24-kube-api-access-jp4wd\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:10.612313 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.612285    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n"
Oct 31 01:23:10.612509 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.612285    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n" event={"ID":"ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf","Type":"ContainerDied","Data":"0d4a078bb7492abb7be0be6745d8429f4dc1791147b2981a9b86ca6447e642e1"}
Oct 31 01:23:10.612509 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.612417    4327 scope.go:122] "RemoveContainer" containerID="64d1de9a8acedbd1fdbedc370885c798ae7ee7fdcab3f106816cfbebc11b52f2"
Oct 31 01:23:10.613868 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.613845    4327 generic.go:356] "Generic (PLEG): container finished" podID="edf4f19d-74fd-4f83-96d2-6c66d711fa24" containerID="54ca17c793ec754eb9d63059cf6fc1879105de9643defc7d2b65fbdd5616818a" exitCode=0
Oct 31 01:23:10.613954 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.613895    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k"
Oct 31 01:23:10.613954 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.613930    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k" event={"ID":"edf4f19d-74fd-4f83-96d2-6c66d711fa24","Type":"ContainerDied","Data":"54ca17c793ec754eb9d63059cf6fc1879105de9643defc7d2b65fbdd5616818a"}
Oct 31 01:23:10.614065 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.613984    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k" event={"ID":"edf4f19d-74fd-4f83-96d2-6c66d711fa24","Type":"ContainerDied","Data":"62add354ee07531f379ed3a76c5e8ee651f6c7212699a2ed4ab9db80a5be5ab1"}
Oct 31 01:23:10.615828 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.615806    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"
Oct 31 01:23:10.615828 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.615804    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j" event={"ID":"2387e9d0-6540-45b2-b624-78f6f63e26f5","Type":"ContainerDied","Data":"919c6dd148f8ace35807f83985567167639c5054292b4c4d344c90c9aaf5bdaf"}
Oct 31 01:23:10.617429 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.617390    4327 generic.go:356] "Generic (PLEG): container finished" podID="ceab7625-0443-4896-a735-5e917167771a" containerID="624c37fbb148f3926a087d15392e4b6e3f0b14c956f58b1916f9c093dce70d8d" exitCode=0
Oct 31 01:23:10.617429 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.617418    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g" event={"ID":"ceab7625-0443-4896-a735-5e917167771a","Type":"ContainerDied","Data":"624c37fbb148f3926a087d15392e4b6e3f0b14c956f58b1916f9c093dce70d8d"}
Oct 31 01:23:10.617639 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.617433    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g"
Oct 31 01:23:10.617639 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.617455    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g" event={"ID":"ceab7625-0443-4896-a735-5e917167771a","Type":"ContainerDied","Data":"d766eecf7bdc86e9ef42b44fd1549573b4fa1206c193683e80d9bf40e43cf50a"}
Oct 31 01:23:10.625581 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.625163    4327 scope.go:122] "RemoveContainer" containerID="54ca17c793ec754eb9d63059cf6fc1879105de9643defc7d2b65fbdd5616818a"
Oct 31 01:23:10.635660 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.635627    4327 scope.go:122] "RemoveContainer" containerID="54ca17c793ec754eb9d63059cf6fc1879105de9643defc7d2b65fbdd5616818a"
Oct 31 01:23:10.635985 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:23:10.635965    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54ca17c793ec754eb9d63059cf6fc1879105de9643defc7d2b65fbdd5616818a\": container with ID starting with 54ca17c793ec754eb9d63059cf6fc1879105de9643defc7d2b65fbdd5616818a not found: ID does not exist" containerID="54ca17c793ec754eb9d63059cf6fc1879105de9643defc7d2b65fbdd5616818a"
Oct 31 01:23:10.636079 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.635993    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ca17c793ec754eb9d63059cf6fc1879105de9643defc7d2b65fbdd5616818a"} err="failed to get container status \"54ca17c793ec754eb9d63059cf6fc1879105de9643defc7d2b65fbdd5616818a\": rpc error: code = NotFound desc = could not find container \"54ca17c793ec754eb9d63059cf6fc1879105de9643defc7d2b65fbdd5616818a\": container with ID starting with 54ca17c793ec754eb9d63059cf6fc1879105de9643defc7d2b65fbdd5616818a not found: ID does not exist"
Oct 31 01:23:10.636079 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.636008    4327 scope.go:122] "RemoveContainer" containerID="04cd907891d7c24397e948a3ec04a5c4bac026a2203197a7909ef12f32aad448"
Oct 31 01:23:10.645205 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.645184    4327 scope.go:122] "RemoveContainer" containerID="624c37fbb148f3926a087d15392e4b6e3f0b14c956f58b1916f9c093dce70d8d"
Oct 31 01:23:10.655882 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.655864    4327 scope.go:122] "RemoveContainer" containerID="624c37fbb148f3926a087d15392e4b6e3f0b14c956f58b1916f9c093dce70d8d"
Oct 31 01:23:10.656163 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:23:10.656142    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"624c37fbb148f3926a087d15392e4b6e3f0b14c956f58b1916f9c093dce70d8d\": container with ID starting with 624c37fbb148f3926a087d15392e4b6e3f0b14c956f58b1916f9c093dce70d8d not found: ID does not exist" containerID="624c37fbb148f3926a087d15392e4b6e3f0b14c956f58b1916f9c093dce70d8d"
Oct 31 01:23:10.656244 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.656169    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"624c37fbb148f3926a087d15392e4b6e3f0b14c956f58b1916f9c093dce70d8d"} err="failed to get container status \"624c37fbb148f3926a087d15392e4b6e3f0b14c956f58b1916f9c093dce70d8d\": rpc error: code = NotFound desc = could not find container \"624c37fbb148f3926a087d15392e4b6e3f0b14c956f58b1916f9c093dce70d8d\": container with ID starting with 624c37fbb148f3926a087d15392e4b6e3f0b14c956f58b1916f9c093dce70d8d not found: ID does not exist"
Oct 31 01:23:10.691340 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.691310    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k"]
Oct 31 01:23:10.703518 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.703492    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-7c7856f95-hkk4k"]
Oct 31 01:23:10.740950 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.740922    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g"]
Oct 31 01:23:10.743280 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.743256    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-114c95-7c5799bc88-w2b2g"]
Oct 31 01:23:10.788803 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.788766    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n"]
Oct 31 01:23:10.794738 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.794710    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-114c95-5ffd5ff745-mrd2n"]
Oct 31 01:23:10.838785 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.838753    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"]
Oct 31 01:23:10.851639 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:10.851613    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-tfh7j"]
Oct 31 01:23:11.921556 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:11.921522    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2387e9d0-6540-45b2-b624-78f6f63e26f5" path="/var/lib/kubelet/pods/2387e9d0-6540-45b2-b624-78f6f63e26f5/volumes"
Oct 31 01:23:11.922031 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:11.922015    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceab7625-0443-4896-a735-5e917167771a" path="/var/lib/kubelet/pods/ceab7625-0443-4896-a735-5e917167771a/volumes"
Oct 31 01:23:11.922421 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:11.922407    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edf4f19d-74fd-4f83-96d2-6c66d711fa24" path="/var/lib/kubelet/pods/edf4f19d-74fd-4f83-96d2-6c66d711fa24/volumes"
Oct 31 01:23:11.923255 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:11.923238    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf" path="/var/lib/kubelet/pods/ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf/volumes"
Oct 31 01:23:12.282258 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.282230    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc"
Oct 31 01:23:12.313289 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.313264    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb"
Oct 31 01:23:12.366874 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.366836    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d05d663-30be-4ee3-97c8-801ab8d0f885-kube-api-access-rz8nl\" (UniqueName: \"kubernetes.io/projected/0d05d663-30be-4ee3-97c8-801ab8d0f885-kube-api-access-rz8nl\") pod \"0d05d663-30be-4ee3-97c8-801ab8d0f885\" (UID: \"0d05d663-30be-4ee3-97c8-801ab8d0f885\") "
Oct 31 01:23:12.367071 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.367006    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c725e03-2ab7-49a9-8251-606dc8f70d4f-kube-api-access-7rb5t\" (UniqueName: \"kubernetes.io/projected/2c725e03-2ab7-49a9-8251-606dc8f70d4f-kube-api-access-7rb5t\") pod \"2c725e03-2ab7-49a9-8251-606dc8f70d4f\" (UID: \"2c725e03-2ab7-49a9-8251-606dc8f70d4f\") "
Oct 31 01:23:12.367144 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.367077    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c725e03-2ab7-49a9-8251-606dc8f70d4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c725e03-2ab7-49a9-8251-606dc8f70d4f-plugins-config-volume\") pod \"2c725e03-2ab7-49a9-8251-606dc8f70d4f\" (UID: \"2c725e03-2ab7-49a9-8251-606dc8f70d4f\") "
Oct 31 01:23:12.367199 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.367186    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d05d663-30be-4ee3-97c8-801ab8d0f885-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d05d663-30be-4ee3-97c8-801ab8d0f885-plugins-config-volume\") pod \"0d05d663-30be-4ee3-97c8-801ab8d0f885\" (UID: \"0d05d663-30be-4ee3-97c8-801ab8d0f885\") "
Oct 31 01:23:12.367565 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.367535    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d05d663-30be-4ee3-97c8-801ab8d0f885-plugins-config-volume" pod "0d05d663-30be-4ee3-97c8-801ab8d0f885" (UID: "0d05d663-30be-4ee3-97c8-801ab8d0f885"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:12.367565 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.367542    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c725e03-2ab7-49a9-8251-606dc8f70d4f-plugins-config-volume" pod "2c725e03-2ab7-49a9-8251-606dc8f70d4f" (UID: "2c725e03-2ab7-49a9-8251-606dc8f70d4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:12.369210 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.369189    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c725e03-2ab7-49a9-8251-606dc8f70d4f-kube-api-access-7rb5t" pod "2c725e03-2ab7-49a9-8251-606dc8f70d4f" (UID: "2c725e03-2ab7-49a9-8251-606dc8f70d4f"). InnerVolumeSpecName "kube-api-access-7rb5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:12.369569 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.369544    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d05d663-30be-4ee3-97c8-801ab8d0f885-kube-api-access-rz8nl" pod "0d05d663-30be-4ee3-97c8-801ab8d0f885" (UID: "0d05d663-30be-4ee3-97c8-801ab8d0f885"). InnerVolumeSpecName "kube-api-access-rz8nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:12.469239 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.469150    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c725e03-2ab7-49a9-8251-606dc8f70d4f-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:12.469239 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.469184    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d05d663-30be-4ee3-97c8-801ab8d0f885-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:12.469239 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.469194    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz8nl\" (UniqueName: \"kubernetes.io/projected/0d05d663-30be-4ee3-97c8-801ab8d0f885-kube-api-access-rz8nl\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:12.469239 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.469203    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rb5t\" (UniqueName: \"kubernetes.io/projected/2c725e03-2ab7-49a9-8251-606dc8f70d4f-kube-api-access-7rb5t\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:12.635875 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.635842    4327 generic.go:356] "Generic (PLEG): container finished" podID="2c725e03-2ab7-49a9-8251-606dc8f70d4f" containerID="451ebdddadb823ba6ff721b3971825b3fa8263929be2fd584f61216bb89a239d" exitCode=0
Oct 31 01:23:12.636069 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.635896    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb"
Oct 31 01:23:12.636069 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.635925    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb" event={"ID":"2c725e03-2ab7-49a9-8251-606dc8f70d4f","Type":"ContainerDied","Data":"451ebdddadb823ba6ff721b3971825b3fa8263929be2fd584f61216bb89a239d"}
Oct 31 01:23:12.636069 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.636005    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb" event={"ID":"2c725e03-2ab7-49a9-8251-606dc8f70d4f","Type":"ContainerDied","Data":"0aa4ccf6d8e49633568b98e56749ee067f2719c930c37b0005ea70ad3d96b073"}
Oct 31 01:23:12.636069 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.636028    4327 scope.go:122] "RemoveContainer" containerID="451ebdddadb823ba6ff721b3971825b3fa8263929be2fd584f61216bb89a239d"
Oct 31 01:23:12.637560 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.637533    4327 generic.go:356] "Generic (PLEG): container finished" podID="0d05d663-30be-4ee3-97c8-801ab8d0f885" containerID="2e284c3bc5b4cfb9abf67cd2588e747e26e5d0406d1c088f8dc926484dc13127" exitCode=0
Oct 31 01:23:12.637716 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.637566    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc" event={"ID":"0d05d663-30be-4ee3-97c8-801ab8d0f885","Type":"ContainerDied","Data":"2e284c3bc5b4cfb9abf67cd2588e747e26e5d0406d1c088f8dc926484dc13127"}
Oct 31 01:23:12.637716 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.637611    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc"
Oct 31 01:23:12.637716 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.637616    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc" event={"ID":"0d05d663-30be-4ee3-97c8-801ab8d0f885","Type":"ContainerDied","Data":"b118eb8e194cf1ff04e04cf64b34f07a3751df09047346755cb26b039d015fec"}
Oct 31 01:23:12.645953 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.645930    4327 scope.go:122] "RemoveContainer" containerID="451ebdddadb823ba6ff721b3971825b3fa8263929be2fd584f61216bb89a239d"
Oct 31 01:23:12.646332 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:23:12.646289    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"451ebdddadb823ba6ff721b3971825b3fa8263929be2fd584f61216bb89a239d\": container with ID starting with 451ebdddadb823ba6ff721b3971825b3fa8263929be2fd584f61216bb89a239d not found: ID does not exist" containerID="451ebdddadb823ba6ff721b3971825b3fa8263929be2fd584f61216bb89a239d"
Oct 31 01:23:12.646471 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.646341    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"451ebdddadb823ba6ff721b3971825b3fa8263929be2fd584f61216bb89a239d"} err="failed to get container status \"451ebdddadb823ba6ff721b3971825b3fa8263929be2fd584f61216bb89a239d\": rpc error: code = NotFound desc = could not find container \"451ebdddadb823ba6ff721b3971825b3fa8263929be2fd584f61216bb89a239d\": container with ID starting with 451ebdddadb823ba6ff721b3971825b3fa8263929be2fd584f61216bb89a239d not found: ID does not exist"
Oct 31 01:23:12.646471 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.646360    4327 scope.go:122] "RemoveContainer" containerID="2e284c3bc5b4cfb9abf67cd2588e747e26e5d0406d1c088f8dc926484dc13127"
Oct 31 01:23:12.656225 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.656207    4327 scope.go:122] "RemoveContainer" containerID="2e284c3bc5b4cfb9abf67cd2588e747e26e5d0406d1c088f8dc926484dc13127"
Oct 31 01:23:12.656457 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:23:12.656439    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e284c3bc5b4cfb9abf67cd2588e747e26e5d0406d1c088f8dc926484dc13127\": container with ID starting with 2e284c3bc5b4cfb9abf67cd2588e747e26e5d0406d1c088f8dc926484dc13127 not found: ID does not exist" containerID="2e284c3bc5b4cfb9abf67cd2588e747e26e5d0406d1c088f8dc926484dc13127"
Oct 31 01:23:12.656510 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.656464    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e284c3bc5b4cfb9abf67cd2588e747e26e5d0406d1c088f8dc926484dc13127"} err="failed to get container status \"2e284c3bc5b4cfb9abf67cd2588e747e26e5d0406d1c088f8dc926484dc13127\": rpc error: code = NotFound desc = could not find container \"2e284c3bc5b4cfb9abf67cd2588e747e26e5d0406d1c088f8dc926484dc13127\": container with ID starting with 2e284c3bc5b4cfb9abf67cd2588e747e26e5d0406d1c088f8dc926484dc13127 not found: ID does not exist"
Oct 31 01:23:12.693065 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.693033    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc"]
Oct 31 01:23:12.699123 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.699093    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ada0c1-7548dd6f85-hkqgc"]
Oct 31 01:23:12.734263 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.734174    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb"]
Oct 31 01:23:12.744279 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:12.744247    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ada0c1-6fd9b86dc5-j8pzb"]
Oct 31 01:23:13.920853 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:13.920820    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d05d663-30be-4ee3-97c8-801ab8d0f885" path="/var/lib/kubelet/pods/0d05d663-30be-4ee3-97c8-801ab8d0f885/volumes"
Oct 31 01:23:13.921251 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:13.921236    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c725e03-2ab7-49a9-8251-606dc8f70d4f" path="/var/lib/kubelet/pods/2c725e03-2ab7-49a9-8251-606dc8f70d4f/volumes"
Oct 31 01:23:15.293444 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:23:15.293400    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:23:34.063538 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:34.063498    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl"]
Oct 31 01:23:34.064207 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:34.063804    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl" podUID="a33d24c7-5486-4dba-aca9-83bedd6b6464" containerName="maas-api" containerID="cri-o://45640486e52c2c898971be8df68d850c7c3d4f94300da678ef7fd2f5e7c2865a" gracePeriod=30
Oct 31 01:23:34.349919 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:34.349891    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl"
Oct 31 01:23:34.446907 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:34.446872    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a33d24c7-5486-4dba-aca9-83bedd6b6464-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a33d24c7-5486-4dba-aca9-83bedd6b6464-maas-api-tls\") pod \"a33d24c7-5486-4dba-aca9-83bedd6b6464\" (UID: \"a33d24c7-5486-4dba-aca9-83bedd6b6464\") "
Oct 31 01:23:34.447111 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:34.447000    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a33d24c7-5486-4dba-aca9-83bedd6b6464-kube-api-access-tftcx\" (UniqueName: \"kubernetes.io/projected/a33d24c7-5486-4dba-aca9-83bedd6b6464-kube-api-access-tftcx\") pod \"a33d24c7-5486-4dba-aca9-83bedd6b6464\" (UID: \"a33d24c7-5486-4dba-aca9-83bedd6b6464\") "
Oct 31 01:23:34.447111 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:34.447044    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a33d24c7-5486-4dba-aca9-83bedd6b6464-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a33d24c7-5486-4dba-aca9-83bedd6b6464-openshift-service-ca\") pod \"a33d24c7-5486-4dba-aca9-83bedd6b6464\" (UID: \"a33d24c7-5486-4dba-aca9-83bedd6b6464\") "
Oct 31 01:23:34.447248 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:34.447160    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a33d24c7-5486-4dba-aca9-83bedd6b6464-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a33d24c7-5486-4dba-aca9-83bedd6b6464-metrics-certs\") pod \"a33d24c7-5486-4dba-aca9-83bedd6b6464\" (UID: \"a33d24c7-5486-4dba-aca9-83bedd6b6464\") "
Oct 31 01:23:34.447536 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:34.447503    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33d24c7-5486-4dba-aca9-83bedd6b6464-openshift-service-ca" pod "a33d24c7-5486-4dba-aca9-83bedd6b6464" (UID: "a33d24c7-5486-4dba-aca9-83bedd6b6464"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:34.447704 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:34.447632    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a33d24c7-5486-4dba-aca9-83bedd6b6464-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:34.449252 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:34.449218    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a33d24c7-5486-4dba-aca9-83bedd6b6464-metrics-certs" pod "a33d24c7-5486-4dba-aca9-83bedd6b6464" (UID: "a33d24c7-5486-4dba-aca9-83bedd6b6464"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:23:34.449389 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:34.449262    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a33d24c7-5486-4dba-aca9-83bedd6b6464-maas-api-tls" pod "a33d24c7-5486-4dba-aca9-83bedd6b6464" (UID: "a33d24c7-5486-4dba-aca9-83bedd6b6464"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:23:34.449389 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:34.449273    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33d24c7-5486-4dba-aca9-83bedd6b6464-kube-api-access-tftcx" pod "a33d24c7-5486-4dba-aca9-83bedd6b6464" (UID: "a33d24c7-5486-4dba-aca9-83bedd6b6464"). InnerVolumeSpecName "kube-api-access-tftcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:34.549053 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:34.548965    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a33d24c7-5486-4dba-aca9-83bedd6b6464-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:34.549053 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:34.549010    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a33d24c7-5486-4dba-aca9-83bedd6b6464-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:34.549053 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:34.549025    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tftcx\" (UniqueName: \"kubernetes.io/projected/a33d24c7-5486-4dba-aca9-83bedd6b6464-kube-api-access-tftcx\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:34.771703 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:34.770767    4327 generic.go:356] "Generic (PLEG): container finished" podID="a33d24c7-5486-4dba-aca9-83bedd6b6464" containerID="45640486e52c2c898971be8df68d850c7c3d4f94300da678ef7fd2f5e7c2865a" exitCode=0
Oct 31 01:23:34.771703 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:34.770824    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl"
Oct 31 01:23:34.771703 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:34.770946    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl" event={"ID":"a33d24c7-5486-4dba-aca9-83bedd6b6464","Type":"ContainerDied","Data":"45640486e52c2c898971be8df68d850c7c3d4f94300da678ef7fd2f5e7c2865a"}
Oct 31 01:23:34.771703 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:34.770998    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl" event={"ID":"a33d24c7-5486-4dba-aca9-83bedd6b6464","Type":"ContainerDied","Data":"82fee2f7326a8723f5a29d2ceaabd38e7a66476a7566987c484102cc41de3767"}
Oct 31 01:23:34.771703 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:34.771027    4327 scope.go:122] "RemoveContainer" containerID="45640486e52c2c898971be8df68d850c7c3d4f94300da678ef7fd2f5e7c2865a"
Oct 31 01:23:34.785932 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:34.785851    4327 scope.go:122] "RemoveContainer" containerID="45640486e52c2c898971be8df68d850c7c3d4f94300da678ef7fd2f5e7c2865a"
Oct 31 01:23:34.786270 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:23:34.786233    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45640486e52c2c898971be8df68d850c7c3d4f94300da678ef7fd2f5e7c2865a\": container with ID starting with 45640486e52c2c898971be8df68d850c7c3d4f94300da678ef7fd2f5e7c2865a not found: ID does not exist" containerID="45640486e52c2c898971be8df68d850c7c3d4f94300da678ef7fd2f5e7c2865a"
Oct 31 01:23:34.786401 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:34.786276    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45640486e52c2c898971be8df68d850c7c3d4f94300da678ef7fd2f5e7c2865a"} err="failed to get container status \"45640486e52c2c898971be8df68d850c7c3d4f94300da678ef7fd2f5e7c2865a\": rpc error: code = NotFound desc = could not find container \"45640486e52c2c898971be8df68d850c7c3d4f94300da678ef7fd2f5e7c2865a\": container with ID starting with 45640486e52c2c898971be8df68d850c7c3d4f94300da678ef7fd2f5e7c2865a not found: ID does not exist"
Oct 31 01:23:34.839003 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:34.838966    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl"]
Oct 31 01:23:34.870946 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:34.870901    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f0029-56b6cf9944-5dqhl"]
Oct 31 01:23:35.921494 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:35.921458    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a33d24c7-5486-4dba-aca9-83bedd6b6464" path="/var/lib/kubelet/pods/a33d24c7-5486-4dba-aca9-83bedd6b6464/volumes"
Oct 31 01:23:36.220673 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:36.220591    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw"]
Oct 31 01:23:36.255448 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:36.255410    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc"]
Oct 31 01:23:41.221384 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:41.221321    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw" podUID="cf5a254c-034c-4179-9e52-f3db86dbfd7e" containerName="payload-processing" containerID="cri-o://54d4778fba44fe6e38fe4b370e05221c76b88185d21b2e28e73a93c030c310d6" gracePeriod=25
Oct 31 01:23:41.255776 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:41.255731    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc" podUID="495d81aa-af2c-4a7c-832d-61dfb5f7531c" containerName="payload-pre-processing" containerID="cri-o://48db15f68c8bd634f2543fa35c81b973abf9a34d6833f61bd16b47540e2131ee" gracePeriod=25
Oct 31 01:23:42.767244 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:42.767195    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"]
Oct 31 01:23:42.767779 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:42.767474    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q" podUID="da54fda6-be77-4967-8c98-8b1f99beeba6" containerName="istio-proxy" containerID="cri-o://e8b38a3ee69bf4a5efbae4bf28e355f7da6cc6b93dfffe6c42f60a9e8119bf67" gracePeriod=30
Oct 31 01:23:42.779336 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:42.779306    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"]
Oct 31 01:23:42.779571 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:42.779545    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5" podUID="41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f" containerName="istio-proxy" containerID="cri-o://8757d9ce7e27cc54551cc354516e10fbba4b1e5f2e0b1cf13f4b514911f9d2d0" gracePeriod=30
Oct 31 01:23:42.790505 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:42.790477    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"]
Oct 31 01:23:42.790697 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:42.790662    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm" podUID="5c4e63bc-59e0-4eeb-85ba-fdc28515aeff" containerName="istio-proxy" containerID="cri-o://76c6b1d4f8dd44035e2ff9670a893e481bb03bce08718173b55149d500df7b3a" gracePeriod=30
Oct 31 01:23:42.909005 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:42.908923    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw" podUID="cf5a254c-034c-4179-9e52-f3db86dbfd7e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:23:42.909862 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:42.909824    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc" podUID="495d81aa-af2c-4a7c-832d-61dfb5f7531c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:23:43.640775 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:43.640729    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm"]
Oct 31 01:23:43.640998 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:43.640970    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm" podUID="39aa5706-7923-41cb-a27e-e2034f07e687" containerName="maas-api" containerID="cri-o://0667c6a7ead02aa8c4a323adcffa340dbeef39917631fbd5533f858f20ea9661" gracePeriod=30
Oct 31 01:23:43.827756 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:43.827720    4327 generic.go:356] "Generic (PLEG): container finished" podID="39aa5706-7923-41cb-a27e-e2034f07e687" containerID="0667c6a7ead02aa8c4a323adcffa340dbeef39917631fbd5533f858f20ea9661" exitCode=0
Oct 31 01:23:43.828233 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:43.827828    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm" event={"ID":"39aa5706-7923-41cb-a27e-e2034f07e687","Type":"ContainerDied","Data":"0667c6a7ead02aa8c4a323adcffa340dbeef39917631fbd5533f858f20ea9661"}
Oct 31 01:23:43.918225 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:43.918193    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm"
Oct 31 01:23:44.027096 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:44.027058    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39aa5706-7923-41cb-a27e-e2034f07e687-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39aa5706-7923-41cb-a27e-e2034f07e687-maas-api-tls\") pod \"39aa5706-7923-41cb-a27e-e2034f07e687\" (UID: \"39aa5706-7923-41cb-a27e-e2034f07e687\") "
Oct 31 01:23:44.027308 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:44.027174    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39aa5706-7923-41cb-a27e-e2034f07e687-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39aa5706-7923-41cb-a27e-e2034f07e687-openshift-service-ca\") pod \"39aa5706-7923-41cb-a27e-e2034f07e687\" (UID: \"39aa5706-7923-41cb-a27e-e2034f07e687\") "
Oct 31 01:23:44.027308 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:44.027225    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39aa5706-7923-41cb-a27e-e2034f07e687-kube-api-access-ld2km\" (UniqueName: \"kubernetes.io/projected/39aa5706-7923-41cb-a27e-e2034f07e687-kube-api-access-ld2km\") pod \"39aa5706-7923-41cb-a27e-e2034f07e687\" (UID: \"39aa5706-7923-41cb-a27e-e2034f07e687\") "
Oct 31 01:23:44.027308 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:44.027262    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39aa5706-7923-41cb-a27e-e2034f07e687-metrics-certs\" (UniqueName: \"kubernetes.io/secret/39aa5706-7923-41cb-a27e-e2034f07e687-metrics-certs\") pod \"39aa5706-7923-41cb-a27e-e2034f07e687\" (UID: \"39aa5706-7923-41cb-a27e-e2034f07e687\") "
Oct 31 01:23:44.027901 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:44.027864    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39aa5706-7923-41cb-a27e-e2034f07e687-openshift-service-ca" pod "39aa5706-7923-41cb-a27e-e2034f07e687" (UID: "39aa5706-7923-41cb-a27e-e2034f07e687"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:44.029932 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:44.029626    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39aa5706-7923-41cb-a27e-e2034f07e687-maas-api-tls" pod "39aa5706-7923-41cb-a27e-e2034f07e687" (UID: "39aa5706-7923-41cb-a27e-e2034f07e687"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:23:44.029932 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:44.029850    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39aa5706-7923-41cb-a27e-e2034f07e687-metrics-certs" pod "39aa5706-7923-41cb-a27e-e2034f07e687" (UID: "39aa5706-7923-41cb-a27e-e2034f07e687"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:23:44.030352 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:44.030324    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39aa5706-7923-41cb-a27e-e2034f07e687-kube-api-access-ld2km" pod "39aa5706-7923-41cb-a27e-e2034f07e687" (UID: "39aa5706-7923-41cb-a27e-e2034f07e687"). InnerVolumeSpecName "kube-api-access-ld2km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:44.128627 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:44.128592    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39aa5706-7923-41cb-a27e-e2034f07e687-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:44.128627 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:44.128623    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld2km\" (UniqueName: \"kubernetes.io/projected/39aa5706-7923-41cb-a27e-e2034f07e687-kube-api-access-ld2km\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:44.128627 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:44.128633    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39aa5706-7923-41cb-a27e-e2034f07e687-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:44.128920 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:44.128641    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39aa5706-7923-41cb-a27e-e2034f07e687-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:44.835293 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:44.835211    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm"
Oct 31 01:23:44.835726 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:44.835215    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm" event={"ID":"39aa5706-7923-41cb-a27e-e2034f07e687","Type":"ContainerDied","Data":"8fa793bd1f459703de945dcb43a06e02153b09fa44916f658c34eea4435a67d7"}
Oct 31 01:23:44.835726 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:44.835351    4327 scope.go:122] "RemoveContainer" containerID="0667c6a7ead02aa8c4a323adcffa340dbeef39917631fbd5533f858f20ea9661"
Oct 31 01:23:44.888490 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:44.888459    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm"]
Oct 31 01:23:44.901082 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:44.901056    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-328d61-9df5b9564-f7wkm"]
Oct 31 01:23:45.292260 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:23:45.292221    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:23:45.920366 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:45.920330    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39aa5706-7923-41cb-a27e-e2034f07e687" path="/var/lib/kubelet/pods/39aa5706-7923-41cb-a27e-e2034f07e687/volumes"
Oct 31 01:23:48.071711 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.069193    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:23:48.100061 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.100031    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:23:48.154306 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.154274    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:48.179600 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.179561    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-workload-certs\") pod \"da54fda6-be77-4967-8c98-8b1f99beeba6\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") "
Oct 31 01:23:48.179835 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.179623    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-workload-socket\") pod \"da54fda6-be77-4967-8c98-8b1f99beeba6\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") "
Oct 31 01:23:48.179835 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.179661    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-token\" (UniqueName: \"kubernetes.io/projected/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-token\") pod \"da54fda6-be77-4967-8c98-8b1f99beeba6\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") "
Oct 31 01:23:48.179835 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.179729    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-kube-api-access-l9vfd\" (UniqueName: \"kubernetes.io/projected/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-kube-api-access-l9vfd\") pod \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") "
Oct 31 01:23:48.179835 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.179773    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da54fda6-be77-4967-8c98-8b1f99beeba6-kube-api-access-lgtgl\" (UniqueName: \"kubernetes.io/projected/da54fda6-be77-4967-8c98-8b1f99beeba6-kube-api-access-lgtgl\") pod \"da54fda6-be77-4967-8c98-8b1f99beeba6\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") "
Oct 31 01:23:48.179835 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.179813    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-workload-socket\") pod \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") "
Oct 31 01:23:48.180829 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.179880    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-podinfo\") pod \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") "
Oct 31 01:23:48.180829 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.179917    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-token\" (UniqueName: \"kubernetes.io/projected/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-token\") pod \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") "
Oct 31 01:23:48.180829 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.179953    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-envoy\") pod \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") "
Oct 31 01:23:48.180829 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.179994    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-podinfo\") pod \"da54fda6-be77-4967-8c98-8b1f99beeba6\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") "
Oct 31 01:23:48.180829 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.180036    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da54fda6-be77-4967-8c98-8b1f99beeba6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da54fda6-be77-4967-8c98-8b1f99beeba6-istiod-ca-cert\") pod \"da54fda6-be77-4967-8c98-8b1f99beeba6\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") "
Oct 31 01:23:48.180829 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.180109    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-workload-certs\") pod \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") "
Oct 31 01:23:48.180829 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.180146    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-data\") pod \"da54fda6-be77-4967-8c98-8b1f99beeba6\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") "
Oct 31 01:23:48.180829 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.180229    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-envoy\") pod \"da54fda6-be77-4967-8c98-8b1f99beeba6\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") "
Oct 31 01:23:48.180829 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.180260    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-credential-socket\") pod \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") "
Oct 31 01:23:48.180829 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.180326    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istiod-ca-cert\") pod \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") "
Oct 31 01:23:48.180829 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.180367    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-credential-socket\") pod \"da54fda6-be77-4967-8c98-8b1f99beeba6\" (UID: \"da54fda6-be77-4967-8c98-8b1f99beeba6\") "
Oct 31 01:23:48.180829 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.180392    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-data\") pod \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\" (UID: \"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f\") "
Oct 31 01:23:48.184884 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.181208    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-data" pod "41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f" (UID: "41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:48.184884 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.181436    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-workload-certs" pod "da54fda6-be77-4967-8c98-8b1f99beeba6" (UID: "da54fda6-be77-4967-8c98-8b1f99beeba6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:48.184884 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.181613    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-workload-socket" pod "da54fda6-be77-4967-8c98-8b1f99beeba6" (UID: "da54fda6-be77-4967-8c98-8b1f99beeba6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:48.184884 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.183358    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-credential-socket" pod "41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f" (UID: "41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:48.184884 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.183408    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-workload-socket" pod "41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f" (UID: "41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:48.184884 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.183748    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istiod-ca-cert" pod "41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f" (UID: "41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:48.184884 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.183753    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-credential-socket" pod "da54fda6-be77-4967-8c98-8b1f99beeba6" (UID: "da54fda6-be77-4967-8c98-8b1f99beeba6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:48.184884 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.183953    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-workload-certs" pod "41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f" (UID: "41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:48.184884 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.183970    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-data" pod "da54fda6-be77-4967-8c98-8b1f99beeba6" (UID: "da54fda6-be77-4967-8c98-8b1f99beeba6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:48.184884 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.184261    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da54fda6-be77-4967-8c98-8b1f99beeba6-istiod-ca-cert" pod "da54fda6-be77-4967-8c98-8b1f99beeba6" (UID: "da54fda6-be77-4967-8c98-8b1f99beeba6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:48.186113 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.186085    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-kube-api-access-l9vfd" pod "41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f" (UID: "41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f"). InnerVolumeSpecName "kube-api-access-l9vfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:48.186223 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.186211    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da54fda6-be77-4967-8c98-8b1f99beeba6-kube-api-access-lgtgl" pod "da54fda6-be77-4967-8c98-8b1f99beeba6" (UID: "da54fda6-be77-4967-8c98-8b1f99beeba6"). InnerVolumeSpecName "kube-api-access-lgtgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:48.187437 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.187414    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-envoy" pod "da54fda6-be77-4967-8c98-8b1f99beeba6" (UID: "da54fda6-be77-4967-8c98-8b1f99beeba6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:48.187527 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.187510    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-podinfo" pod "da54fda6-be77-4967-8c98-8b1f99beeba6" (UID: "da54fda6-be77-4967-8c98-8b1f99beeba6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:23:48.187941 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.187916    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-token" pod "da54fda6-be77-4967-8c98-8b1f99beeba6" (UID: "da54fda6-be77-4967-8c98-8b1f99beeba6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:48.189096 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.188991    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-podinfo" pod "41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f" (UID: "41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:23:48.189096 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.189007    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-envoy" pod "41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f" (UID: "41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:48.189096 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.189072    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-token" pod "41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f" (UID: "41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:48.282254 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.282216    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-workload-socket\") pod \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") "
Oct 31 01:23:48.282443 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.282267    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-credential-socket\") pod \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") "
Oct 31 01:23:48.282443 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.282312    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-workload-certs\") pod \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") "
Oct 31 01:23:48.282443 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.282337    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-token\" (UniqueName: \"kubernetes.io/projected/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-token\") pod \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") "
Oct 31 01:23:48.282443 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.282383    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-kube-api-access-ttvg9\" (UniqueName: \"kubernetes.io/projected/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-kube-api-access-ttvg9\") pod \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") "
Oct 31 01:23:48.282443 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.282405    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-podinfo\") pod \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") "
Oct 31 01:23:48.282749 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.282474    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-envoy\") pod \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") "
Oct 31 01:23:48.282749 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.282510    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-data\") pod \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") "
Oct 31 01:23:48.282749 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.282542    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-workload-socket" pod "5c4e63bc-59e0-4eeb-85ba-fdc28515aeff" (UID: "5c4e63bc-59e0-4eeb-85ba-fdc28515aeff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:48.282749 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.282578    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istiod-ca-cert\") pod \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\" (UID: \"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff\") "
Oct 31 01:23:48.282749 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.282605    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-credential-socket" pod "5c4e63bc-59e0-4eeb-85ba-fdc28515aeff" (UID: "5c4e63bc-59e0-4eeb-85ba-fdc28515aeff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:48.282749 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.282736    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-workload-certs" pod "5c4e63bc-59e0-4eeb-85ba-fdc28515aeff" (UID: "5c4e63bc-59e0-4eeb-85ba-fdc28515aeff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:48.283106 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.282956    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-data" pod "5c4e63bc-59e0-4eeb-85ba-fdc28515aeff" (UID: "5c4e63bc-59e0-4eeb-85ba-fdc28515aeff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:48.283106 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.283037    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istiod-ca-cert" pod "5c4e63bc-59e0-4eeb-85ba-fdc28515aeff" (UID: "5c4e63bc-59e0-4eeb-85ba-fdc28515aeff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:48.283228 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.283145    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.283228 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.283166    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.283228 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.283180    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.283228 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.283192    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.283228 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.283205    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9vfd\" (UniqueName: \"kubernetes.io/projected/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-kube-api-access-l9vfd\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.283228 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.283219    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.283562 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.283233    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgtgl\" (UniqueName: \"kubernetes.io/projected/da54fda6-be77-4967-8c98-8b1f99beeba6-kube-api-access-lgtgl\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.283562 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.283261    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.283562 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.283275    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.283562 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.283288    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.283562 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.283300    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.283562 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.283313    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.283562 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.283325    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.283562 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.283337    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.283562 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.283351    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da54fda6-be77-4967-8c98-8b1f99beeba6-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.283562 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.283364    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.283562 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.283376    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.283562 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.283389    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.283562 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.283403    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.283562 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.283416    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.283562 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.283429    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.283562 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.283442    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da54fda6-be77-4967-8c98-8b1f99beeba6-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.285035 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.285004    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-podinfo" pod "5c4e63bc-59e0-4eeb-85ba-fdc28515aeff" (UID: "5c4e63bc-59e0-4eeb-85ba-fdc28515aeff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:23:48.285139 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.285085    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-envoy" pod "5c4e63bc-59e0-4eeb-85ba-fdc28515aeff" (UID: "5c4e63bc-59e0-4eeb-85ba-fdc28515aeff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:48.285332 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.285312    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-kube-api-access-ttvg9" pod "5c4e63bc-59e0-4eeb-85ba-fdc28515aeff" (UID: "5c4e63bc-59e0-4eeb-85ba-fdc28515aeff"). InnerVolumeSpecName "kube-api-access-ttvg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:48.285409 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.285387    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-token" pod "5c4e63bc-59e0-4eeb-85ba-fdc28515aeff" (UID: "5c4e63bc-59e0-4eeb-85ba-fdc28515aeff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:48.384248 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.384207    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.384248 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.384241    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttvg9\" (UniqueName: \"kubernetes.io/projected/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-kube-api-access-ttvg9\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.384248 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.384256    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.384509 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.384269    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.384509 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.384281    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:48.864916 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.864878    4327 generic.go:356] "Generic (PLEG): container finished" podID="da54fda6-be77-4967-8c98-8b1f99beeba6" containerID="e8b38a3ee69bf4a5efbae4bf28e355f7da6cc6b93dfffe6c42f60a9e8119bf67" exitCode=0
Oct 31 01:23:48.865078 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.864953    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"
Oct 31 01:23:48.865078 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.864956    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q" event={"ID":"da54fda6-be77-4967-8c98-8b1f99beeba6","Type":"ContainerDied","Data":"e8b38a3ee69bf4a5efbae4bf28e355f7da6cc6b93dfffe6c42f60a9e8119bf67"}
Oct 31 01:23:48.865078 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.865049    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q" event={"ID":"da54fda6-be77-4967-8c98-8b1f99beeba6","Type":"ContainerDied","Data":"aec4500db618022927378ac134204660ea3f41cbff0b4d7a59c8848036eab154"}
Oct 31 01:23:48.865235 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.865082    4327 scope.go:122] "RemoveContainer" containerID="e8b38a3ee69bf4a5efbae4bf28e355f7da6cc6b93dfffe6c42f60a9e8119bf67"
Oct 31 01:23:48.866600 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.866577    4327 generic.go:356] "Generic (PLEG): container finished" podID="5c4e63bc-59e0-4eeb-85ba-fdc28515aeff" containerID="76c6b1d4f8dd44035e2ff9670a893e481bb03bce08718173b55149d500df7b3a" exitCode=0
Oct 31 01:23:48.866741 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.866654    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"
Oct 31 01:23:48.866741 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.866665    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm" event={"ID":"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff","Type":"ContainerDied","Data":"76c6b1d4f8dd44035e2ff9670a893e481bb03bce08718173b55149d500df7b3a"}
Oct 31 01:23:48.866741 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.866737    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm" event={"ID":"5c4e63bc-59e0-4eeb-85ba-fdc28515aeff","Type":"ContainerDied","Data":"3fea9f0a591b4a15ed35c5e266e2d8f1be4851008c7cfe3ffb2d5d4a13a2fe1d"}
Oct 31 01:23:48.868279 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.868255    4327 generic.go:356] "Generic (PLEG): container finished" podID="41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f" containerID="8757d9ce7e27cc54551cc354516e10fbba4b1e5f2e0b1cf13f4b514911f9d2d0" exitCode=0
Oct 31 01:23:48.868366 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.868312    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"
Oct 31 01:23:48.868366 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.868319    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5" event={"ID":"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f","Type":"ContainerDied","Data":"8757d9ce7e27cc54551cc354516e10fbba4b1e5f2e0b1cf13f4b514911f9d2d0"}
Oct 31 01:23:48.868473 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.868369    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5" event={"ID":"41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f","Type":"ContainerDied","Data":"b5d6bc6f6e1897e0ef7bfc8c1e7aa3f7e2243d57a70ce4e49c7040a83c5e0986"}
Oct 31 01:23:48.875633 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.875600    4327 scope.go:122] "RemoveContainer" containerID="e8b38a3ee69bf4a5efbae4bf28e355f7da6cc6b93dfffe6c42f60a9e8119bf67"
Oct 31 01:23:48.875992 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:23:48.875972    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8b38a3ee69bf4a5efbae4bf28e355f7da6cc6b93dfffe6c42f60a9e8119bf67\": container with ID starting with e8b38a3ee69bf4a5efbae4bf28e355f7da6cc6b93dfffe6c42f60a9e8119bf67 not found: ID does not exist" containerID="e8b38a3ee69bf4a5efbae4bf28e355f7da6cc6b93dfffe6c42f60a9e8119bf67"
Oct 31 01:23:48.876089 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.876007    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8b38a3ee69bf4a5efbae4bf28e355f7da6cc6b93dfffe6c42f60a9e8119bf67"} err="failed to get container status \"e8b38a3ee69bf4a5efbae4bf28e355f7da6cc6b93dfffe6c42f60a9e8119bf67\": rpc error: code = NotFound desc = could not find container \"e8b38a3ee69bf4a5efbae4bf28e355f7da6cc6b93dfffe6c42f60a9e8119bf67\": container with ID starting with e8b38a3ee69bf4a5efbae4bf28e355f7da6cc6b93dfffe6c42f60a9e8119bf67 not found: ID does not exist"
Oct 31 01:23:48.876089 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.876032    4327 scope.go:122] "RemoveContainer" containerID="76c6b1d4f8dd44035e2ff9670a893e481bb03bce08718173b55149d500df7b3a"
Oct 31 01:23:48.886265 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.886241    4327 scope.go:122] "RemoveContainer" containerID="76c6b1d4f8dd44035e2ff9670a893e481bb03bce08718173b55149d500df7b3a"
Oct 31 01:23:48.886523 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:23:48.886504    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76c6b1d4f8dd44035e2ff9670a893e481bb03bce08718173b55149d500df7b3a\": container with ID starting with 76c6b1d4f8dd44035e2ff9670a893e481bb03bce08718173b55149d500df7b3a not found: ID does not exist" containerID="76c6b1d4f8dd44035e2ff9670a893e481bb03bce08718173b55149d500df7b3a"
Oct 31 01:23:48.886578 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.886531    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76c6b1d4f8dd44035e2ff9670a893e481bb03bce08718173b55149d500df7b3a"} err="failed to get container status \"76c6b1d4f8dd44035e2ff9670a893e481bb03bce08718173b55149d500df7b3a\": rpc error: code = NotFound desc = could not find container \"76c6b1d4f8dd44035e2ff9670a893e481bb03bce08718173b55149d500df7b3a\": container with ID starting with 76c6b1d4f8dd44035e2ff9670a893e481bb03bce08718173b55149d500df7b3a not found: ID does not exist"
Oct 31 01:23:48.886578 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.886547    4327 scope.go:122] "RemoveContainer" containerID="8757d9ce7e27cc54551cc354516e10fbba4b1e5f2e0b1cf13f4b514911f9d2d0"
Oct 31 01:23:48.895241 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.895221    4327 scope.go:122] "RemoveContainer" containerID="8757d9ce7e27cc54551cc354516e10fbba4b1e5f2e0b1cf13f4b514911f9d2d0"
Oct 31 01:23:48.895522 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:23:48.895503    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8757d9ce7e27cc54551cc354516e10fbba4b1e5f2e0b1cf13f4b514911f9d2d0\": container with ID starting with 8757d9ce7e27cc54551cc354516e10fbba4b1e5f2e0b1cf13f4b514911f9d2d0 not found: ID does not exist" containerID="8757d9ce7e27cc54551cc354516e10fbba4b1e5f2e0b1cf13f4b514911f9d2d0"
Oct 31 01:23:48.895568 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.895532    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8757d9ce7e27cc54551cc354516e10fbba4b1e5f2e0b1cf13f4b514911f9d2d0"} err="failed to get container status \"8757d9ce7e27cc54551cc354516e10fbba4b1e5f2e0b1cf13f4b514911f9d2d0\": rpc error: code = NotFound desc = could not find container \"8757d9ce7e27cc54551cc354516e10fbba4b1e5f2e0b1cf13f4b514911f9d2d0\": container with ID starting with 8757d9ce7e27cc54551cc354516e10fbba4b1e5f2e0b1cf13f4b514911f9d2d0 not found: ID does not exist"
Oct 31 01:23:48.929851 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.929810    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"]
Oct 31 01:23:48.942322 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.942290    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-tg7l5"]
Oct 31 01:23:48.971850 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.971760    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"]
Oct 31 01:23:48.973905 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:48.973883    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-z648q"]
Oct 31 01:23:49.012005 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:49.011968    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"]
Oct 31 01:23:49.014448 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:49.014423    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ppjcm"]
Oct 31 01:23:49.920742 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:49.920709    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f" path="/var/lib/kubelet/pods/41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f/volumes"
Oct 31 01:23:49.921181 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:49.921167    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c4e63bc-59e0-4eeb-85ba-fdc28515aeff" path="/var/lib/kubelet/pods/5c4e63bc-59e0-4eeb-85ba-fdc28515aeff/volumes"
Oct 31 01:23:49.921570 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:49.921559    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da54fda6-be77-4967-8c98-8b1f99beeba6" path="/var/lib/kubelet/pods/da54fda6-be77-4967-8c98-8b1f99beeba6/volumes"
Oct 31 01:23:52.904217 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:52.904179    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc" podUID="495d81aa-af2c-4a7c-832d-61dfb5f7531c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:23:52.904217 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:52.904197    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw" podUID="cf5a254c-034c-4179-9e52-f3db86dbfd7e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:23:53.761161 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:53.761124    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb"]
Oct 31 01:23:53.796585 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:53.796550    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b"]
Oct 31 01:23:55.045196 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:55.045141    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t"]
Oct 31 01:23:55.045747 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:55.045356    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t" podUID="0c0783ab-8c5d-4d27-9073-e2a9bc8ede71" containerName="maas-api" containerID="cri-o://481c251a423fc07fc30ac8bea402a191de8975cbc6a28b6945b2b9e05e12a790" gracePeriod=30
Oct 31 01:23:55.324625 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:55.324595    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t"
Oct 31 01:23:55.477167 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:55.477129    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-metrics-certs\") pod \"0c0783ab-8c5d-4d27-9073-e2a9bc8ede71\" (UID: \"0c0783ab-8c5d-4d27-9073-e2a9bc8ede71\") "
Oct 31 01:23:55.477167 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:55.477176    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-maas-api-tls\") pod \"0c0783ab-8c5d-4d27-9073-e2a9bc8ede71\" (UID: \"0c0783ab-8c5d-4d27-9073-e2a9bc8ede71\") "
Oct 31 01:23:55.477449 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:55.477202    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-kube-api-access-9qjn8\" (UniqueName: \"kubernetes.io/projected/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-kube-api-access-9qjn8\") pod \"0c0783ab-8c5d-4d27-9073-e2a9bc8ede71\" (UID: \"0c0783ab-8c5d-4d27-9073-e2a9bc8ede71\") "
Oct 31 01:23:55.477449 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:55.477316    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-openshift-service-ca\") pod \"0c0783ab-8c5d-4d27-9073-e2a9bc8ede71\" (UID: \"0c0783ab-8c5d-4d27-9073-e2a9bc8ede71\") "
Oct 31 01:23:55.477865 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:55.477830    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-openshift-service-ca" pod "0c0783ab-8c5d-4d27-9073-e2a9bc8ede71" (UID: "0c0783ab-8c5d-4d27-9073-e2a9bc8ede71"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:55.479880 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:55.479852    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-kube-api-access-9qjn8" pod "0c0783ab-8c5d-4d27-9073-e2a9bc8ede71" (UID: "0c0783ab-8c5d-4d27-9073-e2a9bc8ede71"). InnerVolumeSpecName "kube-api-access-9qjn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:55.479996 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:55.479955    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-maas-api-tls" pod "0c0783ab-8c5d-4d27-9073-e2a9bc8ede71" (UID: "0c0783ab-8c5d-4d27-9073-e2a9bc8ede71"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:23:55.482128 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:55.482105    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-metrics-certs" pod "0c0783ab-8c5d-4d27-9073-e2a9bc8ede71" (UID: "0c0783ab-8c5d-4d27-9073-e2a9bc8ede71"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:23:55.578812 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:55.578717    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:55.578812 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:55.578752    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:55.578812 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:55.578766    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qjn8\" (UniqueName: \"kubernetes.io/projected/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-kube-api-access-9qjn8\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:55.578812 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:55.578778    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:23:55.911483 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:55.911439    4327 generic.go:356] "Generic (PLEG): container finished" podID="0c0783ab-8c5d-4d27-9073-e2a9bc8ede71" containerID="481c251a423fc07fc30ac8bea402a191de8975cbc6a28b6945b2b9e05e12a790" exitCode=0
Oct 31 01:23:55.911665 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:55.911490    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t"
Oct 31 01:23:55.911665 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:55.911517    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t" event={"ID":"0c0783ab-8c5d-4d27-9073-e2a9bc8ede71","Type":"ContainerDied","Data":"481c251a423fc07fc30ac8bea402a191de8975cbc6a28b6945b2b9e05e12a790"}
Oct 31 01:23:55.911665 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:55.911568    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t" event={"ID":"0c0783ab-8c5d-4d27-9073-e2a9bc8ede71","Type":"ContainerDied","Data":"179dfae242c99cf8b55eb99ef156e0a41d4b5e40c427b5ed2138c271f5705af1"}
Oct 31 01:23:55.911665 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:55.911591    4327 scope.go:122] "RemoveContainer" containerID="481c251a423fc07fc30ac8bea402a191de8975cbc6a28b6945b2b9e05e12a790"
Oct 31 01:23:55.922559 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:55.922529    4327 scope.go:122] "RemoveContainer" containerID="481c251a423fc07fc30ac8bea402a191de8975cbc6a28b6945b2b9e05e12a790"
Oct 31 01:23:55.922933 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:23:55.922910    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"481c251a423fc07fc30ac8bea402a191de8975cbc6a28b6945b2b9e05e12a790\": container with ID starting with 481c251a423fc07fc30ac8bea402a191de8975cbc6a28b6945b2b9e05e12a790 not found: ID does not exist" containerID="481c251a423fc07fc30ac8bea402a191de8975cbc6a28b6945b2b9e05e12a790"
Oct 31 01:23:55.923021 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:55.922939    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"481c251a423fc07fc30ac8bea402a191de8975cbc6a28b6945b2b9e05e12a790"} err="failed to get container status \"481c251a423fc07fc30ac8bea402a191de8975cbc6a28b6945b2b9e05e12a790\": rpc error: code = NotFound desc = could not find container \"481c251a423fc07fc30ac8bea402a191de8975cbc6a28b6945b2b9e05e12a790\": container with ID starting with 481c251a423fc07fc30ac8bea402a191de8975cbc6a28b6945b2b9e05e12a790 not found: ID does not exist"
Oct 31 01:23:55.963812 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:55.963780    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t"]
Oct 31 01:23:55.976166 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:55.976134    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-36af31-c5999b49b-7779t"]
Oct 31 01:23:57.924309 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:57.924268    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c0783ab-8c5d-4d27-9073-e2a9bc8ede71" path="/var/lib/kubelet/pods/0c0783ab-8c5d-4d27-9073-e2a9bc8ede71/volumes"
Oct 31 01:23:58.031258 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:58.031221    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"]
Oct 31 01:23:58.044899 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:58.044827    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"]
Oct 31 01:23:58.048459 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:58.048430    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"]
Oct 31 01:23:58.762239 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:58.762173    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb" podUID="c567b551-e6b1-421a-af55-89097b32d4ff" containerName="payload-processing" containerID="cri-o://d7ebe83d27f827b15ea28cca513c6c19f4447dde7056648355d9c49262b71335" gracePeriod=25
Oct 31 01:23:58.797843 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:23:58.797795    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b" podUID="906ce4bc-762e-461b-99f1-3109cc41c2ff" containerName="payload-pre-processing" containerID="cri-o://b9e88fc672b332fc0b8a2acb94b75dc7a481d85c72db7426438828777e1d7bed" gracePeriod=25
Oct 31 01:24:01.521854 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.521831    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw"
Oct 31 01:24:01.548003 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.547977    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc"
Oct 31 01:24:01.553610 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.553589    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf5a254c-034c-4179-9e52-f3db86dbfd7e-kube-api-access-pv65n\" (UniqueName: \"kubernetes.io/projected/cf5a254c-034c-4179-9e52-f3db86dbfd7e-kube-api-access-pv65n\") pod \"cf5a254c-034c-4179-9e52-f3db86dbfd7e\" (UID: \"cf5a254c-034c-4179-9e52-f3db86dbfd7e\") "
Oct 31 01:24:01.553783 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.553767    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf5a254c-034c-4179-9e52-f3db86dbfd7e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5a254c-034c-4179-9e52-f3db86dbfd7e-plugins-config-volume\") pod \"cf5a254c-034c-4179-9e52-f3db86dbfd7e\" (UID: \"cf5a254c-034c-4179-9e52-f3db86dbfd7e\") "
Oct 31 01:24:01.554103 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.554080    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf5a254c-034c-4179-9e52-f3db86dbfd7e-plugins-config-volume" pod "cf5a254c-034c-4179-9e52-f3db86dbfd7e" (UID: "cf5a254c-034c-4179-9e52-f3db86dbfd7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:01.555645 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.555624    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5a254c-034c-4179-9e52-f3db86dbfd7e-kube-api-access-pv65n" pod "cf5a254c-034c-4179-9e52-f3db86dbfd7e" (UID: "cf5a254c-034c-4179-9e52-f3db86dbfd7e"). InnerVolumeSpecName "kube-api-access-pv65n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:01.654892 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.654856    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/495d81aa-af2c-4a7c-832d-61dfb5f7531c-kube-api-access-qj9vc\" (UniqueName: \"kubernetes.io/projected/495d81aa-af2c-4a7c-832d-61dfb5f7531c-kube-api-access-qj9vc\") pod \"495d81aa-af2c-4a7c-832d-61dfb5f7531c\" (UID: \"495d81aa-af2c-4a7c-832d-61dfb5f7531c\") "
Oct 31 01:24:01.655383 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.654956    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/495d81aa-af2c-4a7c-832d-61dfb5f7531c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/495d81aa-af2c-4a7c-832d-61dfb5f7531c-plugins-config-volume\") pod \"495d81aa-af2c-4a7c-832d-61dfb5f7531c\" (UID: \"495d81aa-af2c-4a7c-832d-61dfb5f7531c\") "
Oct 31 01:24:01.655383 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.655190    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5a254c-034c-4179-9e52-f3db86dbfd7e-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:01.655383 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.655205    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv65n\" (UniqueName: \"kubernetes.io/projected/cf5a254c-034c-4179-9e52-f3db86dbfd7e-kube-api-access-pv65n\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:01.655383 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.655293    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/495d81aa-af2c-4a7c-832d-61dfb5f7531c-plugins-config-volume" pod "495d81aa-af2c-4a7c-832d-61dfb5f7531c" (UID: "495d81aa-af2c-4a7c-832d-61dfb5f7531c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:01.657093 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.657073    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495d81aa-af2c-4a7c-832d-61dfb5f7531c-kube-api-access-qj9vc" pod "495d81aa-af2c-4a7c-832d-61dfb5f7531c" (UID: "495d81aa-af2c-4a7c-832d-61dfb5f7531c"). InnerVolumeSpecName "kube-api-access-qj9vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:01.756213 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.756122    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/495d81aa-af2c-4a7c-832d-61dfb5f7531c-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:01.756213 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.756158    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj9vc\" (UniqueName: \"kubernetes.io/projected/495d81aa-af2c-4a7c-832d-61dfb5f7531c-kube-api-access-qj9vc\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:01.948626 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.948593    4327 generic.go:356] "Generic (PLEG): container finished" podID="495d81aa-af2c-4a7c-832d-61dfb5f7531c" containerID="48db15f68c8bd634f2543fa35c81b973abf9a34d6833f61bd16b47540e2131ee" exitCode=0
Oct 31 01:24:01.948806 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.948648    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc"
Oct 31 01:24:01.948806 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.948689    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc" event={"ID":"495d81aa-af2c-4a7c-832d-61dfb5f7531c","Type":"ContainerDied","Data":"48db15f68c8bd634f2543fa35c81b973abf9a34d6833f61bd16b47540e2131ee"}
Oct 31 01:24:01.948806 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.948760    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc" event={"ID":"495d81aa-af2c-4a7c-832d-61dfb5f7531c","Type":"ContainerDied","Data":"6f06699ade3342c9643ff8c02d6e0717f893de0a8fa0e0845c90512816f43f8e"}
Oct 31 01:24:01.948806 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.948795    4327 scope.go:122] "RemoveContainer" containerID="48db15f68c8bd634f2543fa35c81b973abf9a34d6833f61bd16b47540e2131ee"
Oct 31 01:24:01.950150 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.950127    4327 generic.go:356] "Generic (PLEG): container finished" podID="cf5a254c-034c-4179-9e52-f3db86dbfd7e" containerID="54d4778fba44fe6e38fe4b370e05221c76b88185d21b2e28e73a93c030c310d6" exitCode=0
Oct 31 01:24:01.950265 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.950177    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw"
Oct 31 01:24:01.950265 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.950215    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw" event={"ID":"cf5a254c-034c-4179-9e52-f3db86dbfd7e","Type":"ContainerDied","Data":"54d4778fba44fe6e38fe4b370e05221c76b88185d21b2e28e73a93c030c310d6"}
Oct 31 01:24:01.950394 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.950265    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw" event={"ID":"cf5a254c-034c-4179-9e52-f3db86dbfd7e","Type":"ContainerDied","Data":"b7eb315a24db6f0e81d50f468599551c329345a970d5fccff4980ade9ef848ae"}
Oct 31 01:24:01.958318 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.958295    4327 scope.go:122] "RemoveContainer" containerID="48db15f68c8bd634f2543fa35c81b973abf9a34d6833f61bd16b47540e2131ee"
Oct 31 01:24:01.958595 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:24:01.958570    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48db15f68c8bd634f2543fa35c81b973abf9a34d6833f61bd16b47540e2131ee\": container with ID starting with 48db15f68c8bd634f2543fa35c81b973abf9a34d6833f61bd16b47540e2131ee not found: ID does not exist" containerID="48db15f68c8bd634f2543fa35c81b973abf9a34d6833f61bd16b47540e2131ee"
Oct 31 01:24:01.958650 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.958599    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48db15f68c8bd634f2543fa35c81b973abf9a34d6833f61bd16b47540e2131ee"} err="failed to get container status \"48db15f68c8bd634f2543fa35c81b973abf9a34d6833f61bd16b47540e2131ee\": rpc error: code = NotFound desc = could not find container \"48db15f68c8bd634f2543fa35c81b973abf9a34d6833f61bd16b47540e2131ee\": container with ID starting with 48db15f68c8bd634f2543fa35c81b973abf9a34d6833f61bd16b47540e2131ee not found: ID does not exist"
Oct 31 01:24:01.958650 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.958617    4327 scope.go:122] "RemoveContainer" containerID="54d4778fba44fe6e38fe4b370e05221c76b88185d21b2e28e73a93c030c310d6"
Oct 31 01:24:01.967370 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.967343    4327 scope.go:122] "RemoveContainer" containerID="54d4778fba44fe6e38fe4b370e05221c76b88185d21b2e28e73a93c030c310d6"
Oct 31 01:24:01.967649 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:24:01.967630    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54d4778fba44fe6e38fe4b370e05221c76b88185d21b2e28e73a93c030c310d6\": container with ID starting with 54d4778fba44fe6e38fe4b370e05221c76b88185d21b2e28e73a93c030c310d6 not found: ID does not exist" containerID="54d4778fba44fe6e38fe4b370e05221c76b88185d21b2e28e73a93c030c310d6"
Oct 31 01:24:01.967716 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:01.967656    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d4778fba44fe6e38fe4b370e05221c76b88185d21b2e28e73a93c030c310d6"} err="failed to get container status \"54d4778fba44fe6e38fe4b370e05221c76b88185d21b2e28e73a93c030c310d6\": rpc error: code = NotFound desc = could not find container \"54d4778fba44fe6e38fe4b370e05221c76b88185d21b2e28e73a93c030c310d6\": container with ID starting with 54d4778fba44fe6e38fe4b370e05221c76b88185d21b2e28e73a93c030c310d6 not found: ID does not exist"
Oct 31 01:24:02.000661 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:02.000631    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc"]
Oct 31 01:24:02.012998 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:02.012919    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-328d61-5d597c4b5f-6sfgc"]
Oct 31 01:24:02.043094 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:02.043046    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw"]
Oct 31 01:24:02.045807 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:02.045781    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-328d61-74b757865c-zqvlw"]
Oct 31 01:24:03.033600 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:03.033565    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"]
Oct 31 01:24:03.034149 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:03.033779    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd" podUID="abe1cf54-bbbc-4c89-ba1a-06caf4bcd789" containerName="istio-proxy" containerID="cri-o://ec34be9c4d862024b0bd3a17202ce963c416f9cf11fbd0b066a5fa48b2cedf9b" gracePeriod=30
Oct 31 01:24:03.920074 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:03.920038    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="495d81aa-af2c-4a7c-832d-61dfb5f7531c" path="/var/lib/kubelet/pods/495d81aa-af2c-4a7c-832d-61dfb5f7531c/volumes"
Oct 31 01:24:03.920460 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:03.920447    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf5a254c-034c-4179-9e52-f3db86dbfd7e" path="/var/lib/kubelet/pods/cf5a254c-034c-4179-9e52-f3db86dbfd7e/volumes"
Oct 31 01:24:04.198100 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:04.198002    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl"]
Oct 31 01:24:04.985784 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:04.985743    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b" podUID="906ce4bc-762e-461b-99f1-3109cc41c2ff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:24:06.186398 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:06.186359    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb" podUID="c567b551-e6b1-421a-af55-89097b32d4ff" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:24:07.288167 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:07.288124    4327 patch_prober.go:29] interesting pod/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:24:07.288572 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:07.288199    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd" podUID="abe1cf54-bbbc-4c89-ba1a-06caf4bcd789" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:24:08.301710 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.301632    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:24:08.427417 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.427377    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-podinfo\") pod \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") "
Oct 31 01:24:08.427608 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.427442    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-data\") pod \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") "
Oct 31 01:24:08.427608 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.427473    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-kube-api-access-9snw8\" (UniqueName: \"kubernetes.io/projected/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-kube-api-access-9snw8\") pod \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") "
Oct 31 01:24:08.427608 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.427506    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-credential-socket\") pod \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") "
Oct 31 01:24:08.427608 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.427540    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istiod-ca-cert\") pod \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") "
Oct 31 01:24:08.427608 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.427570    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-token\" (UniqueName: \"kubernetes.io/projected/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-token\") pod \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") "
Oct 31 01:24:08.427608 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.427601    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-workload-socket\") pod \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") "
Oct 31 01:24:08.427973 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.427743    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-data" pod "abe1cf54-bbbc-4c89-ba1a-06caf4bcd789" (UID: "abe1cf54-bbbc-4c89-ba1a-06caf4bcd789"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:08.427973 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.427804    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-workload-certs\") pod \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") "
Oct 31 01:24:08.427973 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.427857    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-envoy\") pod \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\" (UID: \"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789\") "
Oct 31 01:24:08.427973 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.427932    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-credential-socket" pod "abe1cf54-bbbc-4c89-ba1a-06caf4bcd789" (UID: "abe1cf54-bbbc-4c89-ba1a-06caf4bcd789"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:08.427973 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.427943    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-workload-socket" pod "abe1cf54-bbbc-4c89-ba1a-06caf4bcd789" (UID: "abe1cf54-bbbc-4c89-ba1a-06caf4bcd789"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:08.428256 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.428096    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-workload-certs" pod "abe1cf54-bbbc-4c89-ba1a-06caf4bcd789" (UID: "abe1cf54-bbbc-4c89-ba1a-06caf4bcd789"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:08.428256 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.428167    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istiod-ca-cert" pod "abe1cf54-bbbc-4c89-ba1a-06caf4bcd789" (UID: "abe1cf54-bbbc-4c89-ba1a-06caf4bcd789"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:08.428385 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.428322    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:08.428385 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.428346    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:08.428385 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.428359    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:08.428385 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.428373    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:08.428577 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.428386    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:08.429933 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.429900    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-podinfo" pod "abe1cf54-bbbc-4c89-ba1a-06caf4bcd789" (UID: "abe1cf54-bbbc-4c89-ba1a-06caf4bcd789"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:24:08.430288 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.430263    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-envoy" pod "abe1cf54-bbbc-4c89-ba1a-06caf4bcd789" (UID: "abe1cf54-bbbc-4c89-ba1a-06caf4bcd789"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:08.430420 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.430403    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-token" pod "abe1cf54-bbbc-4c89-ba1a-06caf4bcd789" (UID: "abe1cf54-bbbc-4c89-ba1a-06caf4bcd789"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:08.430477 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.430459    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-kube-api-access-9snw8" pod "abe1cf54-bbbc-4c89-ba1a-06caf4bcd789" (UID: "abe1cf54-bbbc-4c89-ba1a-06caf4bcd789"). InnerVolumeSpecName "kube-api-access-9snw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:08.529614 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.529583    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:08.529614 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.529609    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:08.529614 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.529618    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:08.529878 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.529627    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9snw8\" (UniqueName: \"kubernetes.io/projected/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789-kube-api-access-9snw8\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:08.985843 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.985808    4327 generic.go:356] "Generic (PLEG): container finished" podID="abe1cf54-bbbc-4c89-ba1a-06caf4bcd789" containerID="ec34be9c4d862024b0bd3a17202ce963c416f9cf11fbd0b066a5fa48b2cedf9b" exitCode=0
Oct 31 01:24:08.986055 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.985862    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"
Oct 31 01:24:08.986055 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.985906    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd" event={"ID":"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789","Type":"ContainerDied","Data":"ec34be9c4d862024b0bd3a17202ce963c416f9cf11fbd0b066a5fa48b2cedf9b"}
Oct 31 01:24:08.986055 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.985969    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd" event={"ID":"abe1cf54-bbbc-4c89-ba1a-06caf4bcd789","Type":"ContainerDied","Data":"9720f736de1252c1124693b883538b02e180f50df57f8920449493139405bf31"}
Oct 31 01:24:08.986055 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.985998    4327 scope.go:122] "RemoveContainer" containerID="ec34be9c4d862024b0bd3a17202ce963c416f9cf11fbd0b066a5fa48b2cedf9b"
Oct 31 01:24:08.996254 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.996235    4327 scope.go:122] "RemoveContainer" containerID="ec34be9c4d862024b0bd3a17202ce963c416f9cf11fbd0b066a5fa48b2cedf9b"
Oct 31 01:24:08.996567 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:24:08.996547    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec34be9c4d862024b0bd3a17202ce963c416f9cf11fbd0b066a5fa48b2cedf9b\": container with ID starting with ec34be9c4d862024b0bd3a17202ce963c416f9cf11fbd0b066a5fa48b2cedf9b not found: ID does not exist" containerID="ec34be9c4d862024b0bd3a17202ce963c416f9cf11fbd0b066a5fa48b2cedf9b"
Oct 31 01:24:08.996626 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:08.996576    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec34be9c4d862024b0bd3a17202ce963c416f9cf11fbd0b066a5fa48b2cedf9b"} err="failed to get container status \"ec34be9c4d862024b0bd3a17202ce963c416f9cf11fbd0b066a5fa48b2cedf9b\": rpc error: code = NotFound desc = could not find container \"ec34be9c4d862024b0bd3a17202ce963c416f9cf11fbd0b066a5fa48b2cedf9b\": container with ID starting with ec34be9c4d862024b0bd3a17202ce963c416f9cf11fbd0b066a5fa48b2cedf9b not found: ID does not exist"
Oct 31 01:24:09.052220 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:09.052181    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"]
Oct 31 01:24:09.056703 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:09.056666    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-7x6vd"]
Oct 31 01:24:09.198863 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:09.198805    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl" podUID="eb16adc7-12a2-4497-a1a5-ad9adbe96684" containerName="payload-processing" containerID="cri-o://a6998cf36be5fa6d73ae29c7c5738ce83f83d2f104d527ca164742774eefb85f" gracePeriod=25
Oct 31 01:24:09.920233 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:09.920195    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abe1cf54-bbbc-4c89-ba1a-06caf4bcd789" path="/var/lib/kubelet/pods/abe1cf54-bbbc-4c89-ba1a-06caf4bcd789/volumes"
Oct 31 01:24:10.550299 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:10.550262    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl" podUID="eb16adc7-12a2-4497-a1a5-ad9adbe96684" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:24:13.054461 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.054354    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main" containerID="cri-o://5a47e0a6db1aaf5592f4f71c7062a302d32ec8ba8883eac15e7b16e6b146de9d" gracePeriod=45
Oct 31 01:24:13.065547 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.065506    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main" containerID="cri-o://4d06b050052daac737055df47d3866f116fc6da948fd5b2e65afd3ba2d9b17db" gracePeriod=45
Oct 31 01:24:13.068454 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.068414    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" containerID="cri-o://0abd506dbe8103d3eccdc3058f2ec9ee47e528bbf622b167fef7c8a488cfd9c9" gracePeriod=45
Oct 31 01:24:13.341406 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.341378    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 01:24:13.370194 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.370163    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 01:24:13.374129 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.374105    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 01:24:13.482149 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.482119    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/378b2c59-105c-4a30-82cc-4e0d1716414d-kube-api-access-x7cm4\" (UniqueName: \"kubernetes.io/projected/378b2c59-105c-4a30-82cc-4e0d1716414d-kube-api-access-x7cm4\") pod \"378b2c59-105c-4a30-82cc-4e0d1716414d\" (UID: \"378b2c59-105c-4a30-82cc-4e0d1716414d\") "
Oct 31 01:24:13.482331 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.482162    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-model-cache\") pod \"378b2c59-105c-4a30-82cc-4e0d1716414d\" (UID: \"378b2c59-105c-4a30-82cc-4e0d1716414d\") "
Oct 31 01:24:13.482331 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.482183    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-tmp-dir\") pod \"378b2c59-105c-4a30-82cc-4e0d1716414d\" (UID: \"378b2c59-105c-4a30-82cc-4e0d1716414d\") "
Oct 31 01:24:13.482331 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.482215    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-home\" (UniqueName: \"kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-home\") pod \"6deba2aa-764d-4325-8c5d-c09fed312bb6\" (UID: \"6deba2aa-764d-4325-8c5d-c09fed312bb6\") "
Oct 31 01:24:13.482497 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.482331    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6deba2aa-764d-4325-8c5d-c09fed312bb6-tls-certs\" (UniqueName: \"kubernetes.io/secret/6deba2aa-764d-4325-8c5d-c09fed312bb6-tls-certs\") pod \"6deba2aa-764d-4325-8c5d-c09fed312bb6\" (UID: \"6deba2aa-764d-4325-8c5d-c09fed312bb6\") "
Oct 31 01:24:13.482497 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.482375    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-dshm\") pod \"6deba2aa-764d-4325-8c5d-c09fed312bb6\" (UID: \"6deba2aa-764d-4325-8c5d-c09fed312bb6\") "
Oct 31 01:24:13.482497 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.482420    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/681c6060-3662-4f0e-b768-2414c4750db4-tls-certs\" (UniqueName: \"kubernetes.io/secret/681c6060-3662-4f0e-b768-2414c4750db4-tls-certs\") pod \"681c6060-3662-4f0e-b768-2414c4750db4\" (UID: \"681c6060-3662-4f0e-b768-2414c4750db4\") "
Oct 31 01:24:13.482497 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.482471    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-home" pod "6deba2aa-764d-4325-8c5d-c09fed312bb6" (UID: "6deba2aa-764d-4325-8c5d-c09fed312bb6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:13.482497 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.482479    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-model-cache" pod "378b2c59-105c-4a30-82cc-4e0d1716414d" (UID: "378b2c59-105c-4a30-82cc-4e0d1716414d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:13.482820 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.482486    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-tmp-dir" pod "378b2c59-105c-4a30-82cc-4e0d1716414d" (UID: "378b2c59-105c-4a30-82cc-4e0d1716414d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:13.482820 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.482485    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-home\" (UniqueName: \"kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-home\") pod \"378b2c59-105c-4a30-82cc-4e0d1716414d\" (UID: \"378b2c59-105c-4a30-82cc-4e0d1716414d\") "
Oct 31 01:24:13.482820 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.482559    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/681c6060-3662-4f0e-b768-2414c4750db4-kube-api-access-bfdt8\" (UniqueName: \"kubernetes.io/projected/681c6060-3662-4f0e-b768-2414c4750db4-kube-api-access-bfdt8\") pod \"681c6060-3662-4f0e-b768-2414c4750db4\" (UID: \"681c6060-3662-4f0e-b768-2414c4750db4\") "
Oct 31 01:24:13.482820 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.482604    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-model-cache\") pod \"6deba2aa-764d-4325-8c5d-c09fed312bb6\" (UID: \"6deba2aa-764d-4325-8c5d-c09fed312bb6\") "
Oct 31 01:24:13.482820 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.482665    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-tmp-dir\") pod \"6deba2aa-764d-4325-8c5d-c09fed312bb6\" (UID: \"6deba2aa-764d-4325-8c5d-c09fed312bb6\") "
Oct 31 01:24:13.482820 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.482709    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-home" pod "378b2c59-105c-4a30-82cc-4e0d1716414d" (UID: "378b2c59-105c-4a30-82cc-4e0d1716414d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:13.482820 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.482715    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-dshm\") pod \"378b2c59-105c-4a30-82cc-4e0d1716414d\" (UID: \"378b2c59-105c-4a30-82cc-4e0d1716414d\") "
Oct 31 01:24:13.482820 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.482784    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/378b2c59-105c-4a30-82cc-4e0d1716414d-tls-certs\" (UniqueName: \"kubernetes.io/secret/378b2c59-105c-4a30-82cc-4e0d1716414d-tls-certs\") pod \"378b2c59-105c-4a30-82cc-4e0d1716414d\" (UID: \"378b2c59-105c-4a30-82cc-4e0d1716414d\") "
Oct 31 01:24:13.483259 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.482885    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-model-cache" pod "6deba2aa-764d-4325-8c5d-c09fed312bb6" (UID: "6deba2aa-764d-4325-8c5d-c09fed312bb6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:13.483259 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.482896    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-tmp-dir\") pod \"681c6060-3662-4f0e-b768-2414c4750db4\" (UID: \"681c6060-3662-4f0e-b768-2414c4750db4\") "
Oct 31 01:24:13.483259 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.482958    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-dshm\") pod \"681c6060-3662-4f0e-b768-2414c4750db4\" (UID: \"681c6060-3662-4f0e-b768-2414c4750db4\") "
Oct 31 01:24:13.483259 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.482998    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-model-cache\") pod \"681c6060-3662-4f0e-b768-2414c4750db4\" (UID: \"681c6060-3662-4f0e-b768-2414c4750db4\") "
Oct 31 01:24:13.483259 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.483140    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-home\" (UniqueName: \"kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-home\") pod \"681c6060-3662-4f0e-b768-2414c4750db4\" (UID: \"681c6060-3662-4f0e-b768-2414c4750db4\") "
Oct 31 01:24:13.483259 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.483181    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6deba2aa-764d-4325-8c5d-c09fed312bb6-kube-api-access-5fprx\" (UniqueName: \"kubernetes.io/projected/6deba2aa-764d-4325-8c5d-c09fed312bb6-kube-api-access-5fprx\") pod \"6deba2aa-764d-4325-8c5d-c09fed312bb6\" (UID: \"6deba2aa-764d-4325-8c5d-c09fed312bb6\") "
Oct 31 01:24:13.483569 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.483061    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-tmp-dir" pod "681c6060-3662-4f0e-b768-2414c4750db4" (UID: "681c6060-3662-4f0e-b768-2414c4750db4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:13.483636 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.483601    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-tmp-dir" pod "6deba2aa-764d-4325-8c5d-c09fed312bb6" (UID: "6deba2aa-764d-4325-8c5d-c09fed312bb6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:13.483636 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.483620    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-model-cache" pod "681c6060-3662-4f0e-b768-2414c4750db4" (UID: "681c6060-3662-4f0e-b768-2414c4750db4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:13.483840 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.483818    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-home" pod "681c6060-3662-4f0e-b768-2414c4750db4" (UID: "681c6060-3662-4f0e-b768-2414c4750db4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:13.483985 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.483966    4327 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-home\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:13.484054 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.483990    4327 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-home\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:13.484054 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.484003    4327 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-model-cache\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:13.484054 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.484045    4327 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-tmp-dir\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:13.484227 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.484058    4327 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-tmp-dir\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:13.484227 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.484071    4327 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-model-cache\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:13.484227 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.484083    4327 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-home\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:13.484227 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.484095    4327 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-model-cache\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:13.484227 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.484107    4327 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-tmp-dir\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:13.485631 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.485591    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-dshm" pod "378b2c59-105c-4a30-82cc-4e0d1716414d" (UID: "378b2c59-105c-4a30-82cc-4e0d1716414d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:13.485804 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.485642    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/681c6060-3662-4f0e-b768-2414c4750db4-tls-certs" pod "681c6060-3662-4f0e-b768-2414c4750db4" (UID: "681c6060-3662-4f0e-b768-2414c4750db4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:24:13.486318 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.486273    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/378b2c59-105c-4a30-82cc-4e0d1716414d-tls-certs" pod "378b2c59-105c-4a30-82cc-4e0d1716414d" (UID: "378b2c59-105c-4a30-82cc-4e0d1716414d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:24:13.486318 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.486296    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-dshm" pod "6deba2aa-764d-4325-8c5d-c09fed312bb6" (UID: "6deba2aa-764d-4325-8c5d-c09fed312bb6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:13.486516 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.486493    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6deba2aa-764d-4325-8c5d-c09fed312bb6-kube-api-access-5fprx" pod "6deba2aa-764d-4325-8c5d-c09fed312bb6" (UID: "6deba2aa-764d-4325-8c5d-c09fed312bb6"). InnerVolumeSpecName "kube-api-access-5fprx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:13.486573 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.486534    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/378b2c59-105c-4a30-82cc-4e0d1716414d-kube-api-access-x7cm4" pod "378b2c59-105c-4a30-82cc-4e0d1716414d" (UID: "378b2c59-105c-4a30-82cc-4e0d1716414d"). InnerVolumeSpecName "kube-api-access-x7cm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:13.486573 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.486566    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-dshm" pod "681c6060-3662-4f0e-b768-2414c4750db4" (UID: "681c6060-3662-4f0e-b768-2414c4750db4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:13.486760 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.486737    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6deba2aa-764d-4325-8c5d-c09fed312bb6-tls-certs" pod "6deba2aa-764d-4325-8c5d-c09fed312bb6" (UID: "6deba2aa-764d-4325-8c5d-c09fed312bb6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:24:13.487588 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.487560    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681c6060-3662-4f0e-b768-2414c4750db4-kube-api-access-bfdt8" pod "681c6060-3662-4f0e-b768-2414c4750db4" (UID: "681c6060-3662-4f0e-b768-2414c4750db4"). InnerVolumeSpecName "kube-api-access-bfdt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:13.584699 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.584594    4327 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/681c6060-3662-4f0e-b768-2414c4750db4-tls-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:13.584699 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.584628    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfdt8\" (UniqueName: \"kubernetes.io/projected/681c6060-3662-4f0e-b768-2414c4750db4-kube-api-access-bfdt8\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:13.584699 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.584637    4327 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/378b2c59-105c-4a30-82cc-4e0d1716414d-dshm\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:13.584699 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.584645    4327 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/378b2c59-105c-4a30-82cc-4e0d1716414d-tls-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:13.584699 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.584655    4327 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/681c6060-3662-4f0e-b768-2414c4750db4-dshm\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:13.584699 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.584662    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fprx\" (UniqueName: \"kubernetes.io/projected/6deba2aa-764d-4325-8c5d-c09fed312bb6-kube-api-access-5fprx\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:13.584699 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.584670    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7cm4\" (UniqueName: \"kubernetes.io/projected/378b2c59-105c-4a30-82cc-4e0d1716414d-kube-api-access-x7cm4\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:13.584699 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.584704    4327 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6deba2aa-764d-4325-8c5d-c09fed312bb6-tls-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:13.585079 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:13.584716    4327 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6deba2aa-764d-4325-8c5d-c09fed312bb6-dshm\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:14.015646 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.015606    4327 generic.go:356] "Generic (PLEG): container finished" podID="681c6060-3662-4f0e-b768-2414c4750db4" containerID="0abd506dbe8103d3eccdc3058f2ec9ee47e528bbf622b167fef7c8a488cfd9c9" exitCode=0
Oct 31 01:24:14.015646 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.015643    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"
Oct 31 01:24:14.015927 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.015703    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" event={"ID":"681c6060-3662-4f0e-b768-2414c4750db4","Type":"ContainerDied","Data":"0abd506dbe8103d3eccdc3058f2ec9ee47e528bbf622b167fef7c8a488cfd9c9"}
Oct 31 01:24:14.015927 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.015747    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" event={"ID":"681c6060-3662-4f0e-b768-2414c4750db4","Type":"ContainerDied","Data":"8a63533ce05d72fbe67f689311d5f765c4e38fc8521d7e3a789f021002de4ac3"}
Oct 31 01:24:14.015927 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.015768    4327 scope.go:122] "RemoveContainer" containerID="0abd506dbe8103d3eccdc3058f2ec9ee47e528bbf622b167fef7c8a488cfd9c9"
Oct 31 01:24:14.017421 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.017398    4327 generic.go:356] "Generic (PLEG): container finished" podID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerID="4d06b050052daac737055df47d3866f116fc6da948fd5b2e65afd3ba2d9b17db" exitCode=0
Oct 31 01:24:14.017548 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.017449    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"
Oct 31 01:24:14.017548 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.017485    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" event={"ID":"6deba2aa-764d-4325-8c5d-c09fed312bb6","Type":"ContainerDied","Data":"4d06b050052daac737055df47d3866f116fc6da948fd5b2e65afd3ba2d9b17db"}
Oct 31 01:24:14.017548 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.017530    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5" event={"ID":"6deba2aa-764d-4325-8c5d-c09fed312bb6","Type":"ContainerDied","Data":"0b93fd1243ce26501cc9cf39f23277cb3e94211ef3e386fec04ea56a662d703b"}
Oct 31 01:24:14.019240 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.019213    4327 generic.go:356] "Generic (PLEG): container finished" podID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerID="5a47e0a6db1aaf5592f4f71c7062a302d32ec8ba8883eac15e7b16e6b146de9d" exitCode=0
Oct 31 01:24:14.019384 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.019252    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"
Oct 31 01:24:14.019384 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.019308    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" event={"ID":"378b2c59-105c-4a30-82cc-4e0d1716414d","Type":"ContainerDied","Data":"5a47e0a6db1aaf5592f4f71c7062a302d32ec8ba8883eac15e7b16e6b146de9d"}
Oct 31 01:24:14.019384 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.019332    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv" event={"ID":"378b2c59-105c-4a30-82cc-4e0d1716414d","Type":"ContainerDied","Data":"5427d5fc0404d61b52be1c90eedcb2faa9c48e2812045bf03da2133b4536b645"}
Oct 31 01:24:14.026524 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.026501    4327 scope.go:122] "RemoveContainer" containerID="0abd506dbe8103d3eccdc3058f2ec9ee47e528bbf622b167fef7c8a488cfd9c9"
Oct 31 01:24:14.026900 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:24:14.026868    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0abd506dbe8103d3eccdc3058f2ec9ee47e528bbf622b167fef7c8a488cfd9c9\": container with ID starting with 0abd506dbe8103d3eccdc3058f2ec9ee47e528bbf622b167fef7c8a488cfd9c9 not found: ID does not exist" containerID="0abd506dbe8103d3eccdc3058f2ec9ee47e528bbf622b167fef7c8a488cfd9c9"
Oct 31 01:24:14.027021 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.026909    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0abd506dbe8103d3eccdc3058f2ec9ee47e528bbf622b167fef7c8a488cfd9c9"} err="failed to get container status \"0abd506dbe8103d3eccdc3058f2ec9ee47e528bbf622b167fef7c8a488cfd9c9\": rpc error: code = NotFound desc = could not find container \"0abd506dbe8103d3eccdc3058f2ec9ee47e528bbf622b167fef7c8a488cfd9c9\": container with ID starting with 0abd506dbe8103d3eccdc3058f2ec9ee47e528bbf622b167fef7c8a488cfd9c9 not found: ID does not exist"
Oct 31 01:24:14.027021 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.026933    4327 scope.go:122] "RemoveContainer" containerID="4d06b050052daac737055df47d3866f116fc6da948fd5b2e65afd3ba2d9b17db"
Oct 31 01:24:14.035397 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.035377    4327 scope.go:122] "RemoveContainer" containerID="4d06b050052daac737055df47d3866f116fc6da948fd5b2e65afd3ba2d9b17db"
Oct 31 01:24:14.035737 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:24:14.035715    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d06b050052daac737055df47d3866f116fc6da948fd5b2e65afd3ba2d9b17db\": container with ID starting with 4d06b050052daac737055df47d3866f116fc6da948fd5b2e65afd3ba2d9b17db not found: ID does not exist" containerID="4d06b050052daac737055df47d3866f116fc6da948fd5b2e65afd3ba2d9b17db"
Oct 31 01:24:14.035809 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.035746    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d06b050052daac737055df47d3866f116fc6da948fd5b2e65afd3ba2d9b17db"} err="failed to get container status \"4d06b050052daac737055df47d3866f116fc6da948fd5b2e65afd3ba2d9b17db\": rpc error: code = NotFound desc = could not find container \"4d06b050052daac737055df47d3866f116fc6da948fd5b2e65afd3ba2d9b17db\": container with ID starting with 4d06b050052daac737055df47d3866f116fc6da948fd5b2e65afd3ba2d9b17db not found: ID does not exist"
Oct 31 01:24:14.035809 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.035762    4327 scope.go:122] "RemoveContainer" containerID="5a47e0a6db1aaf5592f4f71c7062a302d32ec8ba8883eac15e7b16e6b146de9d"
Oct 31 01:24:14.044758 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.044740    4327 scope.go:122] "RemoveContainer" containerID="5a47e0a6db1aaf5592f4f71c7062a302d32ec8ba8883eac15e7b16e6b146de9d"
Oct 31 01:24:14.045046 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:24:14.045027    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a47e0a6db1aaf5592f4f71c7062a302d32ec8ba8883eac15e7b16e6b146de9d\": container with ID starting with 5a47e0a6db1aaf5592f4f71c7062a302d32ec8ba8883eac15e7b16e6b146de9d not found: ID does not exist" containerID="5a47e0a6db1aaf5592f4f71c7062a302d32ec8ba8883eac15e7b16e6b146de9d"
Oct 31 01:24:14.045108 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.045053    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a47e0a6db1aaf5592f4f71c7062a302d32ec8ba8883eac15e7b16e6b146de9d"} err="failed to get container status \"5a47e0a6db1aaf5592f4f71c7062a302d32ec8ba8883eac15e7b16e6b146de9d\": rpc error: code = NotFound desc = could not find container \"5a47e0a6db1aaf5592f4f71c7062a302d32ec8ba8883eac15e7b16e6b146de9d\": container with ID starting with 5a47e0a6db1aaf5592f4f71c7062a302d32ec8ba8883eac15e7b16e6b146de9d not found: ID does not exist"
Oct 31 01:24:14.069700 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.069643    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"]
Oct 31 01:24:14.080409 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.080381    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-c9e377/distinct-2-w5-c9e377-kserve-747b6556bf-9j8mv"]
Oct 31 01:24:14.110620 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.110590    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"]
Oct 31 01:24:14.114306 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.114281    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99"]
Oct 31 01:24:14.148342 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.148311    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"]
Oct 31 01:24:14.153085 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.153054    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-c9e377/distinct-w5-c9e377-kserve-579796bcfd-dnnl5"]
Oct 31 01:24:14.307424 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.307339    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/premium-w5-c9e377-kserve-856f7b986c-5sd99" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 01:24:14.985507 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:14.985467    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b" podUID="906ce4bc-762e-461b-99f1-3109cc41c2ff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:24:15.304326 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:24:15.304242    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:24:15.919716 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:15.919664    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" path="/var/lib/kubelet/pods/378b2c59-105c-4a30-82cc-4e0d1716414d/volumes"
Oct 31 01:24:15.920125 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:15.920111    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="681c6060-3662-4f0e-b768-2414c4750db4" path="/var/lib/kubelet/pods/681c6060-3662-4f0e-b768-2414c4750db4/volumes"
Oct 31 01:24:15.920478 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:15.920466    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" path="/var/lib/kubelet/pods/6deba2aa-764d-4325-8c5d-c09fed312bb6/volumes"
Oct 31 01:24:16.185936 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:16.185843    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb" podUID="c567b551-e6b1-421a-af55-89097b32d4ff" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:24:18.299424 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:18.293211    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67"]
Oct 31 01:24:18.299424 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:18.293453    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67" podUID="32310634-11e3-4d27-9fd4-e9c3af71002b" containerName="maas-api" containerID="cri-o://ddfb934e3598b5879d065bbd8ce6618fd88a85d74d1e89d54320a9387ff8341c" gracePeriod=30
Oct 31 01:24:18.585258 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:18.585227    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67"
Oct 31 01:24:18.738538 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:18.738500    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32310634-11e3-4d27-9fd4-e9c3af71002b-kube-api-access-nl7f6\" (UniqueName: \"kubernetes.io/projected/32310634-11e3-4d27-9fd4-e9c3af71002b-kube-api-access-nl7f6\") pod \"32310634-11e3-4d27-9fd4-e9c3af71002b\" (UID: \"32310634-11e3-4d27-9fd4-e9c3af71002b\") "
Oct 31 01:24:18.738748 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:18.738554    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32310634-11e3-4d27-9fd4-e9c3af71002b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32310634-11e3-4d27-9fd4-e9c3af71002b-openshift-service-ca\") pod \"32310634-11e3-4d27-9fd4-e9c3af71002b\" (UID: \"32310634-11e3-4d27-9fd4-e9c3af71002b\") "
Oct 31 01:24:18.738748 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:18.738606    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32310634-11e3-4d27-9fd4-e9c3af71002b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/32310634-11e3-4d27-9fd4-e9c3af71002b-metrics-certs\") pod \"32310634-11e3-4d27-9fd4-e9c3af71002b\" (UID: \"32310634-11e3-4d27-9fd4-e9c3af71002b\") "
Oct 31 01:24:18.738748 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:18.738654    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32310634-11e3-4d27-9fd4-e9c3af71002b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32310634-11e3-4d27-9fd4-e9c3af71002b-maas-api-tls\") pod \"32310634-11e3-4d27-9fd4-e9c3af71002b\" (UID: \"32310634-11e3-4d27-9fd4-e9c3af71002b\") "
Oct 31 01:24:18.739076 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:18.739045    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32310634-11e3-4d27-9fd4-e9c3af71002b-openshift-service-ca" pod "32310634-11e3-4d27-9fd4-e9c3af71002b" (UID: "32310634-11e3-4d27-9fd4-e9c3af71002b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:18.740939 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:18.740911    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32310634-11e3-4d27-9fd4-e9c3af71002b-metrics-certs" pod "32310634-11e3-4d27-9fd4-e9c3af71002b" (UID: "32310634-11e3-4d27-9fd4-e9c3af71002b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:24:18.741153 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:18.741134    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32310634-11e3-4d27-9fd4-e9c3af71002b-maas-api-tls" pod "32310634-11e3-4d27-9fd4-e9c3af71002b" (UID: "32310634-11e3-4d27-9fd4-e9c3af71002b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:24:18.743807 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:18.743786    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32310634-11e3-4d27-9fd4-e9c3af71002b-kube-api-access-nl7f6" pod "32310634-11e3-4d27-9fd4-e9c3af71002b" (UID: "32310634-11e3-4d27-9fd4-e9c3af71002b"). InnerVolumeSpecName "kube-api-access-nl7f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:18.840207 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:18.840169    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl7f6\" (UniqueName: \"kubernetes.io/projected/32310634-11e3-4d27-9fd4-e9c3af71002b-kube-api-access-nl7f6\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:18.840207 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:18.840201    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32310634-11e3-4d27-9fd4-e9c3af71002b-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:18.840207 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:18.840213    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32310634-11e3-4d27-9fd4-e9c3af71002b-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:18.840430 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:18.840223    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32310634-11e3-4d27-9fd4-e9c3af71002b-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:19.036575 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.036543    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb"
Oct 31 01:24:19.052043 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.052011    4327 generic.go:356] "Generic (PLEG): container finished" podID="32310634-11e3-4d27-9fd4-e9c3af71002b" containerID="ddfb934e3598b5879d065bbd8ce6618fd88a85d74d1e89d54320a9387ff8341c" exitCode=0
Oct 31 01:24:19.052226 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.052060    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67"
Oct 31 01:24:19.052226 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.052097    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67" event={"ID":"32310634-11e3-4d27-9fd4-e9c3af71002b","Type":"ContainerDied","Data":"ddfb934e3598b5879d065bbd8ce6618fd88a85d74d1e89d54320a9387ff8341c"}
Oct 31 01:24:19.052226 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.052160    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67" event={"ID":"32310634-11e3-4d27-9fd4-e9c3af71002b","Type":"ContainerDied","Data":"e25238ea0e069c3d2c09c4bb944efc171a590abcfcc4d83a3c1fff719c5db323"}
Oct 31 01:24:19.052226 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.052179    4327 scope.go:122] "RemoveContainer" containerID="ddfb934e3598b5879d065bbd8ce6618fd88a85d74d1e89d54320a9387ff8341c"
Oct 31 01:24:19.054375 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.054347    4327 generic.go:356] "Generic (PLEG): container finished" podID="906ce4bc-762e-461b-99f1-3109cc41c2ff" containerID="b9e88fc672b332fc0b8a2acb94b75dc7a481d85c72db7426438828777e1d7bed" exitCode=0
Oct 31 01:24:19.054486 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.054412    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b" event={"ID":"906ce4bc-762e-461b-99f1-3109cc41c2ff","Type":"ContainerDied","Data":"b9e88fc672b332fc0b8a2acb94b75dc7a481d85c72db7426438828777e1d7bed"}
Oct 31 01:24:19.055886 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.055864    4327 generic.go:356] "Generic (PLEG): container finished" podID="c567b551-e6b1-421a-af55-89097b32d4ff" containerID="d7ebe83d27f827b15ea28cca513c6c19f4447dde7056648355d9c49262b71335" exitCode=0
Oct 31 01:24:19.056008 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.055955    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb" event={"ID":"c567b551-e6b1-421a-af55-89097b32d4ff","Type":"ContainerDied","Data":"d7ebe83d27f827b15ea28cca513c6c19f4447dde7056648355d9c49262b71335"}
Oct 31 01:24:19.056008 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.055998    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb" event={"ID":"c567b551-e6b1-421a-af55-89097b32d4ff","Type":"ContainerDied","Data":"19cfc85da4270797c37cfeb6f306d72ebbcf06cd956d0bc46101f626f2524982"}
Oct 31 01:24:19.056165 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.056067    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb"
Oct 31 01:24:19.062845 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.062630    4327 scope.go:122] "RemoveContainer" containerID="ddfb934e3598b5879d065bbd8ce6618fd88a85d74d1e89d54320a9387ff8341c"
Oct 31 01:24:19.063343 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:24:19.063316    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddfb934e3598b5879d065bbd8ce6618fd88a85d74d1e89d54320a9387ff8341c\": container with ID starting with ddfb934e3598b5879d065bbd8ce6618fd88a85d74d1e89d54320a9387ff8341c not found: ID does not exist" containerID="ddfb934e3598b5879d065bbd8ce6618fd88a85d74d1e89d54320a9387ff8341c"
Oct 31 01:24:19.063452 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.063353    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddfb934e3598b5879d065bbd8ce6618fd88a85d74d1e89d54320a9387ff8341c"} err="failed to get container status \"ddfb934e3598b5879d065bbd8ce6618fd88a85d74d1e89d54320a9387ff8341c\": rpc error: code = NotFound desc = could not find container \"ddfb934e3598b5879d065bbd8ce6618fd88a85d74d1e89d54320a9387ff8341c\": container with ID starting with ddfb934e3598b5879d065bbd8ce6618fd88a85d74d1e89d54320a9387ff8341c not found: ID does not exist"
Oct 31 01:24:19.063452 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.063375    4327 scope.go:122] "RemoveContainer" containerID="d7ebe83d27f827b15ea28cca513c6c19f4447dde7056648355d9c49262b71335"
Oct 31 01:24:19.070020 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.069997    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b"
Oct 31 01:24:19.078527 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.076857    4327 scope.go:122] "RemoveContainer" containerID="d7ebe83d27f827b15ea28cca513c6c19f4447dde7056648355d9c49262b71335"
Oct 31 01:24:19.079083 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:24:19.079052    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ebe83d27f827b15ea28cca513c6c19f4447dde7056648355d9c49262b71335\": container with ID starting with d7ebe83d27f827b15ea28cca513c6c19f4447dde7056648355d9c49262b71335 not found: ID does not exist" containerID="d7ebe83d27f827b15ea28cca513c6c19f4447dde7056648355d9c49262b71335"
Oct 31 01:24:19.079228 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.079096    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ebe83d27f827b15ea28cca513c6c19f4447dde7056648355d9c49262b71335"} err="failed to get container status \"d7ebe83d27f827b15ea28cca513c6c19f4447dde7056648355d9c49262b71335\": rpc error: code = NotFound desc = could not find container \"d7ebe83d27f827b15ea28cca513c6c19f4447dde7056648355d9c49262b71335\": container with ID starting with d7ebe83d27f827b15ea28cca513c6c19f4447dde7056648355d9c49262b71335 not found: ID does not exist"
Oct 31 01:24:19.143637 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.143608    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c567b551-e6b1-421a-af55-89097b32d4ff-kube-api-access-8rtbk\" (UniqueName: \"kubernetes.io/projected/c567b551-e6b1-421a-af55-89097b32d4ff-kube-api-access-8rtbk\") pod \"c567b551-e6b1-421a-af55-89097b32d4ff\" (UID: \"c567b551-e6b1-421a-af55-89097b32d4ff\") "
Oct 31 01:24:19.143815 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.143668    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c567b551-e6b1-421a-af55-89097b32d4ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c567b551-e6b1-421a-af55-89097b32d4ff-plugins-config-volume\") pod \"c567b551-e6b1-421a-af55-89097b32d4ff\" (UID: \"c567b551-e6b1-421a-af55-89097b32d4ff\") "
Oct 31 01:24:19.144039 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.144018    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c567b551-e6b1-421a-af55-89097b32d4ff-plugins-config-volume" pod "c567b551-e6b1-421a-af55-89097b32d4ff" (UID: "c567b551-e6b1-421a-af55-89097b32d4ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:19.145763 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.145743    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c567b551-e6b1-421a-af55-89097b32d4ff-kube-api-access-8rtbk" pod "c567b551-e6b1-421a-af55-89097b32d4ff" (UID: "c567b551-e6b1-421a-af55-89097b32d4ff"). InnerVolumeSpecName "kube-api-access-8rtbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:19.192737 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.192701    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67"]
Oct 31 01:24:19.206291 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.206255    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9e377-7d657f7ddb-pjs67"]
Oct 31 01:24:19.244708 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.244643    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/906ce4bc-762e-461b-99f1-3109cc41c2ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/906ce4bc-762e-461b-99f1-3109cc41c2ff-plugins-config-volume\") pod \"906ce4bc-762e-461b-99f1-3109cc41c2ff\" (UID: \"906ce4bc-762e-461b-99f1-3109cc41c2ff\") "
Oct 31 01:24:19.244893 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.244797    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/906ce4bc-762e-461b-99f1-3109cc41c2ff-kube-api-access-tlfg8\" (UniqueName: \"kubernetes.io/projected/906ce4bc-762e-461b-99f1-3109cc41c2ff-kube-api-access-tlfg8\") pod \"906ce4bc-762e-461b-99f1-3109cc41c2ff\" (UID: \"906ce4bc-762e-461b-99f1-3109cc41c2ff\") "
Oct 31 01:24:19.244995 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.244983    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rtbk\" (UniqueName: \"kubernetes.io/projected/c567b551-e6b1-421a-af55-89097b32d4ff-kube-api-access-8rtbk\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:19.245041 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.244997    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c567b551-e6b1-421a-af55-89097b32d4ff-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:19.245041 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.245017    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/906ce4bc-762e-461b-99f1-3109cc41c2ff-plugins-config-volume" pod "906ce4bc-762e-461b-99f1-3109cc41c2ff" (UID: "906ce4bc-762e-461b-99f1-3109cc41c2ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:19.246876 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.246858    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/906ce4bc-762e-461b-99f1-3109cc41c2ff-kube-api-access-tlfg8" pod "906ce4bc-762e-461b-99f1-3109cc41c2ff" (UID: "906ce4bc-762e-461b-99f1-3109cc41c2ff"). InnerVolumeSpecName "kube-api-access-tlfg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:19.345673 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.345581    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlfg8\" (UniqueName: \"kubernetes.io/projected/906ce4bc-762e-461b-99f1-3109cc41c2ff-kube-api-access-tlfg8\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:19.345673 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.345618    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/906ce4bc-762e-461b-99f1-3109cc41c2ff-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:19.423456 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.423420    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb"]
Oct 31 01:24:19.437377 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.437300    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-36af31-868779fbd5-cqsxb"]
Oct 31 01:24:19.920776 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.920723    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32310634-11e3-4d27-9fd4-e9c3af71002b" path="/var/lib/kubelet/pods/32310634-11e3-4d27-9fd4-e9c3af71002b/volumes"
Oct 31 01:24:19.921342 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:19.921319    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c567b551-e6b1-421a-af55-89097b32d4ff" path="/var/lib/kubelet/pods/c567b551-e6b1-421a-af55-89097b32d4ff/volumes"
Oct 31 01:24:20.063493 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:20.063459    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b"
Oct 31 01:24:20.063493 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:20.063470    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b" event={"ID":"906ce4bc-762e-461b-99f1-3109cc41c2ff","Type":"ContainerDied","Data":"5b5ad490b0bd35d3dbbd612f33a7f34523b8158d8861738f639dde0863a47f7f"}
Oct 31 01:24:20.063810 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:20.063577    4327 scope.go:122] "RemoveContainer" containerID="b9e88fc672b332fc0b8a2acb94b75dc7a481d85c72db7426438828777e1d7bed"
Oct 31 01:24:20.120173 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:20.120137    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b"]
Oct 31 01:24:20.132455 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:20.132428    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-36af31-5f69cb665f-tlc8b"]
Oct 31 01:24:20.550906 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:20.550867    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl" podUID="eb16adc7-12a2-4497-a1a5-ad9adbe96684" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:24:21.919378 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:21.919345    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="906ce4bc-762e-461b-99f1-3109cc41c2ff" path="/var/lib/kubelet/pods/906ce4bc-762e-461b-99f1-3109cc41c2ff/volumes"
Oct 31 01:24:23.702532 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:23.702496    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"]
Oct 31 01:24:23.703059 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:23.702734    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv" podUID="b38a6a95-545c-43c6-a861-645d17ddfab4" containerName="istio-proxy" containerID="cri-o://3f92227528bca418bd077c891ea98b1837153ab7f84661ad8064ac181ee0295b" gracePeriod=30
Oct 31 01:24:26.772316 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:26.772278    4327 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:24:26.772747 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:26.772346    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv" podUID="b38a6a95-545c-43c6-a861-645d17ddfab4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:24:28.947753 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:28.947725    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 01:24:29.111701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.111643    4327 generic.go:356] "Generic (PLEG): container finished" podID="b38a6a95-545c-43c6-a861-645d17ddfab4" containerID="3f92227528bca418bd077c891ea98b1837153ab7f84661ad8064ac181ee0295b" exitCode=0
Oct 31 01:24:29.111894 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.111730    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"
Oct 31 01:24:29.111894 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.111731    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv" event={"ID":"b38a6a95-545c-43c6-a861-645d17ddfab4","Type":"ContainerDied","Data":"3f92227528bca418bd077c891ea98b1837153ab7f84661ad8064ac181ee0295b"}
Oct 31 01:24:29.111894 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.111791    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv" event={"ID":"b38a6a95-545c-43c6-a861-645d17ddfab4","Type":"ContainerDied","Data":"e53e4316201c515c1e181b9034fd2e9f8fa377a9e618cbcaa1402c909e6085d8"}
Oct 31 01:24:29.111894 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.111812    4327 scope.go:122] "RemoveContainer" containerID="3f92227528bca418bd077c891ea98b1837153ab7f84661ad8064ac181ee0295b"
Oct 31 01:24:29.120861 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.120838    4327 scope.go:122] "RemoveContainer" containerID="66b8ef3bdab3fc9045e444361fb0143be82cf98130331ecd99b03ccb08a4c247"
Oct 31 01:24:29.129161 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.129144    4327 scope.go:122] "RemoveContainer" containerID="3f92227528bca418bd077c891ea98b1837153ab7f84661ad8064ac181ee0295b"
Oct 31 01:24:29.129424 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:24:29.129402    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f92227528bca418bd077c891ea98b1837153ab7f84661ad8064ac181ee0295b\": container with ID starting with 3f92227528bca418bd077c891ea98b1837153ab7f84661ad8064ac181ee0295b not found: ID does not exist" containerID="3f92227528bca418bd077c891ea98b1837153ab7f84661ad8064ac181ee0295b"
Oct 31 01:24:29.129509 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.129437    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f92227528bca418bd077c891ea98b1837153ab7f84661ad8064ac181ee0295b"} err="failed to get container status \"3f92227528bca418bd077c891ea98b1837153ab7f84661ad8064ac181ee0295b\": rpc error: code = NotFound desc = could not find container \"3f92227528bca418bd077c891ea98b1837153ab7f84661ad8064ac181ee0295b\": container with ID starting with 3f92227528bca418bd077c891ea98b1837153ab7f84661ad8064ac181ee0295b not found: ID does not exist"
Oct 31 01:24:29.129509 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.129461    4327 scope.go:122] "RemoveContainer" containerID="66b8ef3bdab3fc9045e444361fb0143be82cf98130331ecd99b03ccb08a4c247"
Oct 31 01:24:29.129737 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:24:29.129721    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66b8ef3bdab3fc9045e444361fb0143be82cf98130331ecd99b03ccb08a4c247\": container with ID starting with 66b8ef3bdab3fc9045e444361fb0143be82cf98130331ecd99b03ccb08a4c247 not found: ID does not exist" containerID="66b8ef3bdab3fc9045e444361fb0143be82cf98130331ecd99b03ccb08a4c247"
Oct 31 01:24:29.129805 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.129746    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b8ef3bdab3fc9045e444361fb0143be82cf98130331ecd99b03ccb08a4c247"} err="failed to get container status \"66b8ef3bdab3fc9045e444361fb0143be82cf98130331ecd99b03ccb08a4c247\": rpc error: code = NotFound desc = could not find container \"66b8ef3bdab3fc9045e444361fb0143be82cf98130331ecd99b03ccb08a4c247\": container with ID starting with 66b8ef3bdab3fc9045e444361fb0143be82cf98130331ecd99b03ccb08a4c247 not found: ID does not exist"
Oct 31 01:24:29.136009 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.135991    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-data\") pod \"b38a6a95-545c-43c6-a861-645d17ddfab4\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") "
Oct 31 01:24:29.136096 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.136023    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-credential-socket\") pod \"b38a6a95-545c-43c6-a861-645d17ddfab4\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") "
Oct 31 01:24:29.136154 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.136110    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-workload-certs\") pod \"b38a6a95-545c-43c6-a861-645d17ddfab4\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") "
Oct 31 01:24:29.136217 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.136176    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-workload-socket\") pod \"b38a6a95-545c-43c6-a861-645d17ddfab4\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") "
Oct 31 01:24:29.136281 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.136216    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b38a6a95-545c-43c6-a861-645d17ddfab4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38a6a95-545c-43c6-a861-645d17ddfab4-istiod-ca-cert\") pod \"b38a6a95-545c-43c6-a861-645d17ddfab4\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") "
Oct 31 01:24:29.136281 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.136249    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-envoy\") pod \"b38a6a95-545c-43c6-a861-645d17ddfab4\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") "
Oct 31 01:24:29.136281 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.136264    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-data" pod "b38a6a95-545c-43c6-a861-645d17ddfab4" (UID: "b38a6a95-545c-43c6-a861-645d17ddfab4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:29.136444 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.136311    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b38a6a95-545c-43c6-a861-645d17ddfab4-kube-api-access-rb6f2\" (UniqueName: \"kubernetes.io/projected/b38a6a95-545c-43c6-a861-645d17ddfab4-kube-api-access-rb6f2\") pod \"b38a6a95-545c-43c6-a861-645d17ddfab4\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") "
Oct 31 01:24:29.136444 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.136348    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-podinfo\") pod \"b38a6a95-545c-43c6-a861-645d17ddfab4\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") "
Oct 31 01:24:29.136444 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.136307    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-credential-socket" pod "b38a6a95-545c-43c6-a861-645d17ddfab4" (UID: "b38a6a95-545c-43c6-a861-645d17ddfab4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:29.136444 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.136383    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-token\" (UniqueName: \"kubernetes.io/projected/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-token\") pod \"b38a6a95-545c-43c6-a861-645d17ddfab4\" (UID: \"b38a6a95-545c-43c6-a861-645d17ddfab4\") "
Oct 31 01:24:29.136444 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.136396    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-workload-certs" pod "b38a6a95-545c-43c6-a861-645d17ddfab4" (UID: "b38a6a95-545c-43c6-a861-645d17ddfab4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:29.136708 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.136479    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-workload-socket" pod "b38a6a95-545c-43c6-a861-645d17ddfab4" (UID: "b38a6a95-545c-43c6-a861-645d17ddfab4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:29.136708 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.136621    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b38a6a95-545c-43c6-a861-645d17ddfab4-istiod-ca-cert" pod "b38a6a95-545c-43c6-a861-645d17ddfab4" (UID: "b38a6a95-545c-43c6-a861-645d17ddfab4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:29.136833 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.136762    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:29.136833 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.136783    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:29.136833 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.136795    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:29.136833 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.136803    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:29.136833 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.136811    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38a6a95-545c-43c6-a861-645d17ddfab4-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:29.138493 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.138471    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-token" pod "b38a6a95-545c-43c6-a861-645d17ddfab4" (UID: "b38a6a95-545c-43c6-a861-645d17ddfab4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:29.138614 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.138536    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b38a6a95-545c-43c6-a861-645d17ddfab4-kube-api-access-rb6f2" pod "b38a6a95-545c-43c6-a861-645d17ddfab4" (UID: "b38a6a95-545c-43c6-a861-645d17ddfab4"). InnerVolumeSpecName "kube-api-access-rb6f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:29.138853 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.138831    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-envoy" pod "b38a6a95-545c-43c6-a861-645d17ddfab4" (UID: "b38a6a95-545c-43c6-a861-645d17ddfab4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:29.138959 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.138925    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-podinfo" pod "b38a6a95-545c-43c6-a861-645d17ddfab4" (UID: "b38a6a95-545c-43c6-a861-645d17ddfab4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:24:29.237460 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.237424    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb6f2\" (UniqueName: \"kubernetes.io/projected/b38a6a95-545c-43c6-a861-645d17ddfab4-kube-api-access-rb6f2\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:29.237460 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.237457    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:29.237460 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.237467    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:29.237719 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.237476    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38a6a95-545c-43c6-a861-645d17ddfab4-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:29.440033 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.440006    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl"
Oct 31 01:24:29.460185 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.460155    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"]
Oct 31 01:24:29.472599 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.472573    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-mspbv"]
Oct 31 01:24:29.540189 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.540154    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb16adc7-12a2-4497-a1a5-ad9adbe96684-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb16adc7-12a2-4497-a1a5-ad9adbe96684-plugins-config-volume\") pod \"eb16adc7-12a2-4497-a1a5-ad9adbe96684\" (UID: \"eb16adc7-12a2-4497-a1a5-ad9adbe96684\") "
Oct 31 01:24:29.540376 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.540287    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb16adc7-12a2-4497-a1a5-ad9adbe96684-kube-api-access-p242r\" (UniqueName: \"kubernetes.io/projected/eb16adc7-12a2-4497-a1a5-ad9adbe96684-kube-api-access-p242r\") pod \"eb16adc7-12a2-4497-a1a5-ad9adbe96684\" (UID: \"eb16adc7-12a2-4497-a1a5-ad9adbe96684\") "
Oct 31 01:24:29.540581 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.540546    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb16adc7-12a2-4497-a1a5-ad9adbe96684-plugins-config-volume" pod "eb16adc7-12a2-4497-a1a5-ad9adbe96684" (UID: "eb16adc7-12a2-4497-a1a5-ad9adbe96684"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:29.542768 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.542726    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb16adc7-12a2-4497-a1a5-ad9adbe96684-kube-api-access-p242r" pod "eb16adc7-12a2-4497-a1a5-ad9adbe96684" (UID: "eb16adc7-12a2-4497-a1a5-ad9adbe96684"). InnerVolumeSpecName "kube-api-access-p242r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:29.642087 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.641994    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb16adc7-12a2-4497-a1a5-ad9adbe96684-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:29.642087 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.642035    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p242r\" (UniqueName: \"kubernetes.io/projected/eb16adc7-12a2-4497-a1a5-ad9adbe96684-kube-api-access-p242r\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:29.920166 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:29.920089    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b38a6a95-545c-43c6-a861-645d17ddfab4" path="/var/lib/kubelet/pods/b38a6a95-545c-43c6-a861-645d17ddfab4/volumes"
Oct 31 01:24:30.117619 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:30.117582    4327 generic.go:356] "Generic (PLEG): container finished" podID="eb16adc7-12a2-4497-a1a5-ad9adbe96684" containerID="a6998cf36be5fa6d73ae29c7c5738ce83f83d2f104d527ca164742774eefb85f" exitCode=0
Oct 31 01:24:30.118148 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:30.117628    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl"
Oct 31 01:24:30.118148 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:30.117691    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl" event={"ID":"eb16adc7-12a2-4497-a1a5-ad9adbe96684","Type":"ContainerDied","Data":"a6998cf36be5fa6d73ae29c7c5738ce83f83d2f104d527ca164742774eefb85f"}
Oct 31 01:24:30.118148 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:30.117751    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl" event={"ID":"eb16adc7-12a2-4497-a1a5-ad9adbe96684","Type":"ContainerDied","Data":"3bbafea85e629c12d5e33b8b8f931d2a32810357ac38d147dda59b7d635eb8fc"}
Oct 31 01:24:30.118148 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:30.117785    4327 scope.go:122] "RemoveContainer" containerID="a6998cf36be5fa6d73ae29c7c5738ce83f83d2f104d527ca164742774eefb85f"
Oct 31 01:24:30.126673 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:30.126653    4327 scope.go:122] "RemoveContainer" containerID="a6998cf36be5fa6d73ae29c7c5738ce83f83d2f104d527ca164742774eefb85f"
Oct 31 01:24:30.126968 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:24:30.126947    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6998cf36be5fa6d73ae29c7c5738ce83f83d2f104d527ca164742774eefb85f\": container with ID starting with a6998cf36be5fa6d73ae29c7c5738ce83f83d2f104d527ca164742774eefb85f not found: ID does not exist" containerID="a6998cf36be5fa6d73ae29c7c5738ce83f83d2f104d527ca164742774eefb85f"
Oct 31 01:24:30.127034 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:30.126977    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6998cf36be5fa6d73ae29c7c5738ce83f83d2f104d527ca164742774eefb85f"} err="failed to get container status \"a6998cf36be5fa6d73ae29c7c5738ce83f83d2f104d527ca164742774eefb85f\": rpc error: code = NotFound desc = could not find container \"a6998cf36be5fa6d73ae29c7c5738ce83f83d2f104d527ca164742774eefb85f\": container with ID starting with a6998cf36be5fa6d73ae29c7c5738ce83f83d2f104d527ca164742774eefb85f not found: ID does not exist"
Oct 31 01:24:30.169516 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:30.169487    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl"]
Oct 31 01:24:30.182424 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:30.182360    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-c9e377-5b9f9bd69c-xdrrl"]
Oct 31 01:24:31.918938 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:31.918905    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb16adc7-12a2-4497-a1a5-ad9adbe96684" path="/var/lib/kubelet/pods/eb16adc7-12a2-4497-a1a5-ad9adbe96684/volumes"
Oct 31 01:24:45.292019 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:24:45.291975    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:24:52.764468 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:52.764423    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl"]
Oct 31 01:24:52.765062 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:52.764719    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl" podUID="eb06a1d3-494c-4b1b-bb35-30bbc7b23357" containerName="maas-api" containerID="cri-o://90788b13355f851b3ee6bbcb601ad9f7060abefc8ff5c93ecaced51549db9d2a" gracePeriod=30
Oct 31 01:24:53.027586 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:53.027521    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl"
Oct 31 01:24:53.163134 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:53.163100    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-metrics-certs\") pod \"eb06a1d3-494c-4b1b-bb35-30bbc7b23357\" (UID: \"eb06a1d3-494c-4b1b-bb35-30bbc7b23357\") "
Oct 31 01:24:53.163329 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:53.163155    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-maas-api-tls\") pod \"eb06a1d3-494c-4b1b-bb35-30bbc7b23357\" (UID: \"eb06a1d3-494c-4b1b-bb35-30bbc7b23357\") "
Oct 31 01:24:53.163329 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:53.163186    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-openshift-service-ca\") pod \"eb06a1d3-494c-4b1b-bb35-30bbc7b23357\" (UID: \"eb06a1d3-494c-4b1b-bb35-30bbc7b23357\") "
Oct 31 01:24:53.163329 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:53.163263    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-kube-api-access-jl2cl\" (UniqueName: \"kubernetes.io/projected/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-kube-api-access-jl2cl\") pod \"eb06a1d3-494c-4b1b-bb35-30bbc7b23357\" (UID: \"eb06a1d3-494c-4b1b-bb35-30bbc7b23357\") "
Oct 31 01:24:53.163694 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:53.163658    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-openshift-service-ca" pod "eb06a1d3-494c-4b1b-bb35-30bbc7b23357" (UID: "eb06a1d3-494c-4b1b-bb35-30bbc7b23357"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:53.165537 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:53.165508    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-maas-api-tls" pod "eb06a1d3-494c-4b1b-bb35-30bbc7b23357" (UID: "eb06a1d3-494c-4b1b-bb35-30bbc7b23357"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:24:53.165673 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:53.165514    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-kube-api-access-jl2cl" pod "eb06a1d3-494c-4b1b-bb35-30bbc7b23357" (UID: "eb06a1d3-494c-4b1b-bb35-30bbc7b23357"). InnerVolumeSpecName "kube-api-access-jl2cl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:53.165673 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:53.165593    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-metrics-certs" pod "eb06a1d3-494c-4b1b-bb35-30bbc7b23357" (UID: "eb06a1d3-494c-4b1b-bb35-30bbc7b23357"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:24:53.225350 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:53.225311    4327 generic.go:356] "Generic (PLEG): container finished" podID="eb06a1d3-494c-4b1b-bb35-30bbc7b23357" containerID="90788b13355f851b3ee6bbcb601ad9f7060abefc8ff5c93ecaced51549db9d2a" exitCode=0
Oct 31 01:24:53.225562 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:53.225355    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl" event={"ID":"eb06a1d3-494c-4b1b-bb35-30bbc7b23357","Type":"ContainerDied","Data":"90788b13355f851b3ee6bbcb601ad9f7060abefc8ff5c93ecaced51549db9d2a"}
Oct 31 01:24:53.225562 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:53.225372    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl"
Oct 31 01:24:53.225562 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:53.225409    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl" event={"ID":"eb06a1d3-494c-4b1b-bb35-30bbc7b23357","Type":"ContainerDied","Data":"8b4c935c9746aa14a0e4d839b9a5fc711db54e9c14a31ab6611e8dedcf1c446c"}
Oct 31 01:24:53.225562 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:53.225437    4327 scope.go:122] "RemoveContainer" containerID="90788b13355f851b3ee6bbcb601ad9f7060abefc8ff5c93ecaced51549db9d2a"
Oct 31 01:24:53.235159 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:53.235133    4327 scope.go:122] "RemoveContainer" containerID="90788b13355f851b3ee6bbcb601ad9f7060abefc8ff5c93ecaced51549db9d2a"
Oct 31 01:24:53.235504 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:24:53.235483    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90788b13355f851b3ee6bbcb601ad9f7060abefc8ff5c93ecaced51549db9d2a\": container with ID starting with 90788b13355f851b3ee6bbcb601ad9f7060abefc8ff5c93ecaced51549db9d2a not found: ID does not exist" containerID="90788b13355f851b3ee6bbcb601ad9f7060abefc8ff5c93ecaced51549db9d2a"
Oct 31 01:24:53.235562 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:53.235517    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90788b13355f851b3ee6bbcb601ad9f7060abefc8ff5c93ecaced51549db9d2a"} err="failed to get container status \"90788b13355f851b3ee6bbcb601ad9f7060abefc8ff5c93ecaced51549db9d2a\": rpc error: code = NotFound desc = could not find container \"90788b13355f851b3ee6bbcb601ad9f7060abefc8ff5c93ecaced51549db9d2a\": container with ID starting with 90788b13355f851b3ee6bbcb601ad9f7060abefc8ff5c93ecaced51549db9d2a not found: ID does not exist"
Oct 31 01:24:53.264011 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:53.263981    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:53.264011 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:53.264010    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:53.264228 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:53.264023    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:53.264228 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:53.264036    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl2cl\" (UniqueName: \"kubernetes.io/projected/eb06a1d3-494c-4b1b-bb35-30bbc7b23357-kube-api-access-jl2cl\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:24:53.277299 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:53.277266    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl"]
Oct 31 01:24:53.281692 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:53.281606    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-735123-769bbdf5c6-cqmfl"]
Oct 31 01:24:53.919793 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:24:53.919761    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb06a1d3-494c-4b1b-bb35-30bbc7b23357" path="/var/lib/kubelet/pods/eb06a1d3-494c-4b1b-bb35-30bbc7b23357/volumes"
Oct 31 01:25:00.551631 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:00.551582    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"]
Oct 31 01:25:00.552200 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:00.551883    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp" podUID="f32926f1-4427-4764-b86d-7294eaba9e0f" containerName="istio-proxy" containerID="cri-o://b5a4cc410545249d87358041f2bd130a1805239ce0da144e3bf5def222695c42" gracePeriod=30
Oct 31 01:25:05.444405 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.444366    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9c8d8b4d5-frhlg"]
Oct 31 01:25:05.444877 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.444582    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-9c8d8b4d5-frhlg" podUID="b5773d6f-3b5a-4e78-9d77-858c15e04639" containerName="manager" containerID="cri-o://f3a6e1abd5a80aec35338046a0348d14d2f3b5322577c0196a7f18fd93cdf47c" gracePeriod=30
Oct 31 01:25:05.696809 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.696735    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9c8d8b4d5-frhlg"
Oct 31 01:25:05.783201 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.783161    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5773d6f-3b5a-4e78-9d77-858c15e04639-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5773d6f-3b5a-4e78-9d77-858c15e04639-metrics-certs\") pod \"b5773d6f-3b5a-4e78-9d77-858c15e04639\" (UID: \"b5773d6f-3b5a-4e78-9d77-858c15e04639\") "
Oct 31 01:25:05.783410 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.783219    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5773d6f-3b5a-4e78-9d77-858c15e04639-kube-api-access-ql49d\" (UniqueName: \"kubernetes.io/projected/b5773d6f-3b5a-4e78-9d77-858c15e04639-kube-api-access-ql49d\") pod \"b5773d6f-3b5a-4e78-9d77-858c15e04639\" (UID: \"b5773d6f-3b5a-4e78-9d77-858c15e04639\") "
Oct 31 01:25:05.783410 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.783250    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5773d6f-3b5a-4e78-9d77-858c15e04639-webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5773d6f-3b5a-4e78-9d77-858c15e04639-webhook-cert\") pod \"b5773d6f-3b5a-4e78-9d77-858c15e04639\" (UID: \"b5773d6f-3b5a-4e78-9d77-858c15e04639\") "
Oct 31 01:25:05.786139 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.786087    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5773d6f-3b5a-4e78-9d77-858c15e04639-metrics-certs" pod "b5773d6f-3b5a-4e78-9d77-858c15e04639" (UID: "b5773d6f-3b5a-4e78-9d77-858c15e04639"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:25:05.786347 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.786326    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5773d6f-3b5a-4e78-9d77-858c15e04639-webhook-cert" pod "b5773d6f-3b5a-4e78-9d77-858c15e04639" (UID: "b5773d6f-3b5a-4e78-9d77-858c15e04639"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:25:05.786474 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.786454    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5773d6f-3b5a-4e78-9d77-858c15e04639-kube-api-access-ql49d" pod "b5773d6f-3b5a-4e78-9d77-858c15e04639" (UID: "b5773d6f-3b5a-4e78-9d77-858c15e04639"). InnerVolumeSpecName "kube-api-access-ql49d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:25:05.807267 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.807237    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 01:25:05.883976 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.883936    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-podinfo\") pod \"f32926f1-4427-4764-b86d-7294eaba9e0f\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") "
Oct 31 01:25:05.884185 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.883994    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f32926f1-4427-4764-b86d-7294eaba9e0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32926f1-4427-4764-b86d-7294eaba9e0f-istiod-ca-cert\") pod \"f32926f1-4427-4764-b86d-7294eaba9e0f\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") "
Oct 31 01:25:05.884185 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.884064    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-workload-socket\") pod \"f32926f1-4427-4764-b86d-7294eaba9e0f\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") "
Oct 31 01:25:05.884185 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.884094    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-token\" (UniqueName: \"kubernetes.io/projected/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-token\") pod \"f32926f1-4427-4764-b86d-7294eaba9e0f\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") "
Oct 31 01:25:05.884377 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.884186    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-data\") pod \"f32926f1-4427-4764-b86d-7294eaba9e0f\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") "
Oct 31 01:25:05.884377 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.884233    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f32926f1-4427-4764-b86d-7294eaba9e0f-kube-api-access-5w6vt\" (UniqueName: \"kubernetes.io/projected/f32926f1-4427-4764-b86d-7294eaba9e0f-kube-api-access-5w6vt\") pod \"f32926f1-4427-4764-b86d-7294eaba9e0f\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") "
Oct 31 01:25:05.884377 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.884269    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-envoy\") pod \"f32926f1-4427-4764-b86d-7294eaba9e0f\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") "
Oct 31 01:25:05.884377 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.884317    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-workload-certs\") pod \"f32926f1-4427-4764-b86d-7294eaba9e0f\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") "
Oct 31 01:25:05.884600 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.884379    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-credential-socket\") pod \"f32926f1-4427-4764-b86d-7294eaba9e0f\" (UID: \"f32926f1-4427-4764-b86d-7294eaba9e0f\") "
Oct 31 01:25:05.884660 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.884648    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5773d6f-3b5a-4e78-9d77-858c15e04639-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:25:05.884740 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.884665    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql49d\" (UniqueName: \"kubernetes.io/projected/b5773d6f-3b5a-4e78-9d77-858c15e04639-kube-api-access-ql49d\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:25:05.884803 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.884744    4327 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5773d6f-3b5a-4e78-9d77-858c15e04639-webhook-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:25:05.884984 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.884960    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-credential-socket" pod "f32926f1-4427-4764-b86d-7294eaba9e0f" (UID: "f32926f1-4427-4764-b86d-7294eaba9e0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:25:05.885074 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.884998    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-workload-socket" pod "f32926f1-4427-4764-b86d-7294eaba9e0f" (UID: "f32926f1-4427-4764-b86d-7294eaba9e0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:25:05.885215 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.885180    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-data" pod "f32926f1-4427-4764-b86d-7294eaba9e0f" (UID: "f32926f1-4427-4764-b86d-7294eaba9e0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:25:05.885286 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.885192    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-workload-certs" pod "f32926f1-4427-4764-b86d-7294eaba9e0f" (UID: "f32926f1-4427-4764-b86d-7294eaba9e0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:25:05.885574 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.885550    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f32926f1-4427-4764-b86d-7294eaba9e0f-istiod-ca-cert" pod "f32926f1-4427-4764-b86d-7294eaba9e0f" (UID: "f32926f1-4427-4764-b86d-7294eaba9e0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:25:05.887529 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.887485    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-envoy" pod "f32926f1-4427-4764-b86d-7294eaba9e0f" (UID: "f32926f1-4427-4764-b86d-7294eaba9e0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:25:05.887658 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.887595    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32926f1-4427-4764-b86d-7294eaba9e0f-kube-api-access-5w6vt" pod "f32926f1-4427-4764-b86d-7294eaba9e0f" (UID: "f32926f1-4427-4764-b86d-7294eaba9e0f"). InnerVolumeSpecName "kube-api-access-5w6vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:25:05.887775 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.887662    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-podinfo" pod "f32926f1-4427-4764-b86d-7294eaba9e0f" (UID: "f32926f1-4427-4764-b86d-7294eaba9e0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:25:05.888199 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.888168    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-token" pod "f32926f1-4427-4764-b86d-7294eaba9e0f" (UID: "f32926f1-4427-4764-b86d-7294eaba9e0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:25:05.985862 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.985758    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:25:05.985862 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.985799    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32926f1-4427-4764-b86d-7294eaba9e0f-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:25:05.985862 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.985816    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:25:05.985862 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.985831    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:25:05.985862 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.985845    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:25:05.985862 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.985859    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w6vt\" (UniqueName: \"kubernetes.io/projected/f32926f1-4427-4764-b86d-7294eaba9e0f-kube-api-access-5w6vt\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:25:05.986293 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.985874    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:25:05.986293 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.985891    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:25:05.986293 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:05.985903    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32926f1-4427-4764-b86d-7294eaba9e0f-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:25:06.289588 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:06.289490    4327 generic.go:356] "Generic (PLEG): container finished" podID="f32926f1-4427-4764-b86d-7294eaba9e0f" containerID="b5a4cc410545249d87358041f2bd130a1805239ce0da144e3bf5def222695c42" exitCode=0
Oct 31 01:25:06.289588 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:06.289545    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"
Oct 31 01:25:06.289588 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:06.289569    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp" event={"ID":"f32926f1-4427-4764-b86d-7294eaba9e0f","Type":"ContainerDied","Data":"b5a4cc410545249d87358041f2bd130a1805239ce0da144e3bf5def222695c42"}
Oct 31 01:25:06.289931 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:06.289632    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp" event={"ID":"f32926f1-4427-4764-b86d-7294eaba9e0f","Type":"ContainerDied","Data":"b1a414f638e1bedbe3c43909754f6a532d4920a206f9d7b356f768f4fada2f1a"}
Oct 31 01:25:06.289931 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:06.289656    4327 scope.go:122] "RemoveContainer" containerID="b5a4cc410545249d87358041f2bd130a1805239ce0da144e3bf5def222695c42"
Oct 31 01:25:06.291317 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:06.291294    4327 generic.go:356] "Generic (PLEG): container finished" podID="b5773d6f-3b5a-4e78-9d77-858c15e04639" containerID="f3a6e1abd5a80aec35338046a0348d14d2f3b5322577c0196a7f18fd93cdf47c" exitCode=0
Oct 31 01:25:06.291410 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:06.291344    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9c8d8b4d5-frhlg" event={"ID":"b5773d6f-3b5a-4e78-9d77-858c15e04639","Type":"ContainerDied","Data":"f3a6e1abd5a80aec35338046a0348d14d2f3b5322577c0196a7f18fd93cdf47c"}
Oct 31 01:25:06.291410 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:06.291388    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9c8d8b4d5-frhlg" event={"ID":"b5773d6f-3b5a-4e78-9d77-858c15e04639","Type":"ContainerDied","Data":"3320ad6ebde556cbdd6de83054cf3388b9ee459d51ea769182f51a321dae96ef"}
Oct 31 01:25:06.291515 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:06.291346    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9c8d8b4d5-frhlg"
Oct 31 01:25:06.298601 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:06.298571    4327 scope.go:122] "RemoveContainer" containerID="b5a4cc410545249d87358041f2bd130a1805239ce0da144e3bf5def222695c42"
Oct 31 01:25:06.298915 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:25:06.298884    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5a4cc410545249d87358041f2bd130a1805239ce0da144e3bf5def222695c42\": container with ID starting with b5a4cc410545249d87358041f2bd130a1805239ce0da144e3bf5def222695c42 not found: ID does not exist" containerID="b5a4cc410545249d87358041f2bd130a1805239ce0da144e3bf5def222695c42"
Oct 31 01:25:06.298994 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:06.298925    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5a4cc410545249d87358041f2bd130a1805239ce0da144e3bf5def222695c42"} err="failed to get container status \"b5a4cc410545249d87358041f2bd130a1805239ce0da144e3bf5def222695c42\": rpc error: code = NotFound desc = could not find container \"b5a4cc410545249d87358041f2bd130a1805239ce0da144e3bf5def222695c42\": container with ID starting with b5a4cc410545249d87358041f2bd130a1805239ce0da144e3bf5def222695c42 not found: ID does not exist"
Oct 31 01:25:06.298994 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:06.298942    4327 scope.go:122] "RemoveContainer" containerID="f3a6e1abd5a80aec35338046a0348d14d2f3b5322577c0196a7f18fd93cdf47c"
Oct 31 01:25:06.307865 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:06.307838    4327 scope.go:122] "RemoveContainer" containerID="f3a6e1abd5a80aec35338046a0348d14d2f3b5322577c0196a7f18fd93cdf47c"
Oct 31 01:25:06.308196 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:25:06.308151    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3a6e1abd5a80aec35338046a0348d14d2f3b5322577c0196a7f18fd93cdf47c\": container with ID starting with f3a6e1abd5a80aec35338046a0348d14d2f3b5322577c0196a7f18fd93cdf47c not found: ID does not exist" containerID="f3a6e1abd5a80aec35338046a0348d14d2f3b5322577c0196a7f18fd93cdf47c"
Oct 31 01:25:06.308297 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:06.308199    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3a6e1abd5a80aec35338046a0348d14d2f3b5322577c0196a7f18fd93cdf47c"} err="failed to get container status \"f3a6e1abd5a80aec35338046a0348d14d2f3b5322577c0196a7f18fd93cdf47c\": rpc error: code = NotFound desc = could not find container \"f3a6e1abd5a80aec35338046a0348d14d2f3b5322577c0196a7f18fd93cdf47c\": container with ID starting with f3a6e1abd5a80aec35338046a0348d14d2f3b5322577c0196a7f18fd93cdf47c not found: ID does not exist"
Oct 31 01:25:06.342280 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:06.342244    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9c8d8b4d5-frhlg"]
Oct 31 01:25:06.354630 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:06.354594    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-9c8d8b4d5-frhlg"]
Oct 31 01:25:06.384889 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:06.384839    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"]
Oct 31 01:25:06.388173 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:06.388145    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-cddfp"]
Oct 31 01:25:07.337426 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.337384    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9c8d8b4d5-tfsdd"]
Oct 31 01:25:07.337977 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.337943    4327 cpu_manager.go:414] "containerMap: removing container" podUID="da54fda6-be77-4967-8c98-8b1f99beeba6" containerName="istio-proxy"
Oct 31 01:25:07.337977 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.337964    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da54fda6-be77-4967-8c98-8b1f99beeba6" containerName="istio-proxy"
Oct 31 01:25:07.338098 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.337986    4327 cpu_manager.go:414] "containerMap: removing container" podUID="c853f186-bade-4124-b526-438799578a2c" containerName="payload-processing"
Oct 31 01:25:07.338098 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.337994    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c853f186-bade-4124-b526-438799578a2c" containerName="payload-processing"
Oct 31 01:25:07.338098 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338004    4327 cpu_manager.go:414] "containerMap: removing container" podUID="39aa5706-7923-41cb-a27e-e2034f07e687" containerName="maas-api"
Oct 31 01:25:07.338098 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338013    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39aa5706-7923-41cb-a27e-e2034f07e687" containerName="maas-api"
Oct 31 01:25:07.338098 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338026    4327 cpu_manager.go:414] "containerMap: removing container" podUID="a33d24c7-5486-4dba-aca9-83bedd6b6464" containerName="maas-api"
Oct 31 01:25:07.338098 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338035    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a33d24c7-5486-4dba-aca9-83bedd6b6464" containerName="maas-api"
Oct 31 01:25:07.338098 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338047    4327 cpu_manager.go:414] "containerMap: removing container" podUID="2387e9d0-6540-45b2-b624-78f6f63e26f5" containerName="istio-proxy"
Oct 31 01:25:07.338098 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338056    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2387e9d0-6540-45b2-b624-78f6f63e26f5" containerName="istio-proxy"
Oct 31 01:25:07.338098 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338067    4327 cpu_manager.go:414] "containerMap: removing container" podUID="eb16adc7-12a2-4497-a1a5-ad9adbe96684" containerName="payload-processing"
Oct 31 01:25:07.338098 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338076    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb16adc7-12a2-4497-a1a5-ad9adbe96684" containerName="payload-processing"
Oct 31 01:25:07.338098 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338086    4327 cpu_manager.go:414] "containerMap: removing container" podUID="b5773d6f-3b5a-4e78-9d77-858c15e04639" containerName="manager"
Oct 31 01:25:07.338098 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338093    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5773d6f-3b5a-4e78-9d77-858c15e04639" containerName="manager"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338104    4327 cpu_manager.go:414] "containerMap: removing container" podUID="41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f" containerName="istio-proxy"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338113    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f" containerName="istio-proxy"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338123    4327 cpu_manager.go:414] "containerMap: removing container" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338131    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338150    4327 cpu_manager.go:414] "containerMap: removing container" podUID="9dcfe474-6b38-4ada-9272-a48f011722b3" containerName="payload-pre-processing"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338157    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dcfe474-6b38-4ada-9272-a48f011722b3" containerName="payload-pre-processing"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338171    4327 cpu_manager.go:414] "containerMap: removing container" podUID="32310634-11e3-4d27-9fd4-e9c3af71002b" containerName="maas-api"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338178    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32310634-11e3-4d27-9fd4-e9c3af71002b" containerName="maas-api"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338191    4327 cpu_manager.go:414] "containerMap: removing container" podUID="495d81aa-af2c-4a7c-832d-61dfb5f7531c" containerName="payload-pre-processing"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338199    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="495d81aa-af2c-4a7c-832d-61dfb5f7531c" containerName="payload-pre-processing"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338209    4327 cpu_manager.go:414] "containerMap: removing container" podUID="eb06a1d3-494c-4b1b-bb35-30bbc7b23357" containerName="maas-api"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338216    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb06a1d3-494c-4b1b-bb35-30bbc7b23357" containerName="maas-api"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338227    4327 cpu_manager.go:414] "containerMap: removing container" podUID="b38a6a95-545c-43c6-a861-645d17ddfab4" containerName="istio-proxy"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338233    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b38a6a95-545c-43c6-a861-645d17ddfab4" containerName="istio-proxy"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338244    4327 cpu_manager.go:414] "containerMap: removing container" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338251    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338262    4327 cpu_manager.go:414] "containerMap: removing container" podUID="0c0783ab-8c5d-4d27-9073-e2a9bc8ede71" containerName="maas-api"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338269    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c0783ab-8c5d-4d27-9073-e2a9bc8ede71" containerName="maas-api"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338279    4327 cpu_manager.go:414] "containerMap: removing container" podUID="abe1cf54-bbbc-4c89-ba1a-06caf4bcd789" containerName="istio-proxy"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338287    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abe1cf54-bbbc-4c89-ba1a-06caf4bcd789" containerName="istio-proxy"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338296    4327 cpu_manager.go:414] "containerMap: removing container" podUID="0d05d663-30be-4ee3-97c8-801ab8d0f885" containerName="payload-processing"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338304    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d05d663-30be-4ee3-97c8-801ab8d0f885" containerName="payload-processing"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338314    4327 cpu_manager.go:414] "containerMap: removing container" podUID="b38a6a95-545c-43c6-a861-645d17ddfab4" containerName="istio-proxy"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338322    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b38a6a95-545c-43c6-a861-645d17ddfab4" containerName="istio-proxy"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338335    4327 cpu_manager.go:414] "containerMap: removing container" podUID="cf5a254c-034c-4179-9e52-f3db86dbfd7e" containerName="payload-processing"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338342    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf5a254c-034c-4179-9e52-f3db86dbfd7e" containerName="payload-processing"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338354    4327 cpu_manager.go:414] "containerMap: removing container" podUID="c567b551-e6b1-421a-af55-89097b32d4ff" containerName="payload-processing"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338363    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c567b551-e6b1-421a-af55-89097b32d4ff" containerName="payload-processing"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338376    4327 cpu_manager.go:414] "containerMap: removing container" podUID="ceab7625-0443-4896-a735-5e917167771a" containerName="payload-processing"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338383    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceab7625-0443-4896-a735-5e917167771a" containerName="payload-processing"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338393    4327 cpu_manager.go:414] "containerMap: removing container" podUID="f32926f1-4427-4764-b86d-7294eaba9e0f" containerName="istio-proxy"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338400    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f32926f1-4427-4764-b86d-7294eaba9e0f" containerName="istio-proxy"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338410    4327 cpu_manager.go:414] "containerMap: removing container" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338418    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338433    4327 cpu_manager.go:414] "containerMap: removing container" podUID="906ce4bc-762e-461b-99f1-3109cc41c2ff" containerName="payload-pre-processing"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338441    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="906ce4bc-762e-461b-99f1-3109cc41c2ff" containerName="payload-pre-processing"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338451    4327 cpu_manager.go:414] "containerMap: removing container" podUID="5c4e63bc-59e0-4eeb-85ba-fdc28515aeff" containerName="istio-proxy"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338459    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c4e63bc-59e0-4eeb-85ba-fdc28515aeff" containerName="istio-proxy"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338469    4327 cpu_manager.go:414] "containerMap: removing container" podUID="edf4f19d-74fd-4f83-96d2-6c66d711fa24" containerName="payload-pre-processing"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338476    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edf4f19d-74fd-4f83-96d2-6c66d711fa24" containerName="payload-pre-processing"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338490    4327 cpu_manager.go:414] "containerMap: removing container" podUID="2c725e03-2ab7-49a9-8251-606dc8f70d4f" containerName="payload-pre-processing"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338497    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c725e03-2ab7-49a9-8251-606dc8f70d4f" containerName="payload-pre-processing"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338507    4327 cpu_manager.go:414] "containerMap: removing container" podUID="ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf" containerName="maas-api"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338514    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf" containerName="maas-api"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338612    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="41f5a2fa-d0dd-43f4-ace4-e02a8aa4197f" containerName="istio-proxy"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338624    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="378b2c59-105c-4a30-82cc-4e0d1716414d" containerName="main"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338637    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dcfe474-6b38-4ada-9272-a48f011722b3" containerName="payload-pre-processing"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338648    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="32310634-11e3-4d27-9fd4-e9c3af71002b" containerName="maas-api"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338658    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="495d81aa-af2c-4a7c-832d-61dfb5f7531c" containerName="payload-pre-processing"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338666    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb06a1d3-494c-4b1b-bb35-30bbc7b23357" containerName="maas-api"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338693    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="b38a6a95-545c-43c6-a861-645d17ddfab4" containerName="istio-proxy"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338703    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="681c6060-3662-4f0e-b768-2414c4750db4" containerName="main"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338711    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c0783ab-8c5d-4d27-9073-e2a9bc8ede71" containerName="maas-api"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338720    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="abe1cf54-bbbc-4c89-ba1a-06caf4bcd789" containerName="istio-proxy"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338729    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d05d663-30be-4ee3-97c8-801ab8d0f885" containerName="payload-processing"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338737    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="b38a6a95-545c-43c6-a861-645d17ddfab4" containerName="istio-proxy"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338745    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf5a254c-034c-4179-9e52-f3db86dbfd7e" containerName="payload-processing"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338754    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="c567b551-e6b1-421a-af55-89097b32d4ff" containerName="payload-processing"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338762    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceab7625-0443-4896-a735-5e917167771a" containerName="payload-processing"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338770    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="f32926f1-4427-4764-b86d-7294eaba9e0f" containerName="istio-proxy"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338779    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="6deba2aa-764d-4325-8c5d-c09fed312bb6" containerName="main"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338787    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="906ce4bc-762e-461b-99f1-3109cc41c2ff" containerName="payload-pre-processing"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338796    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c4e63bc-59e0-4eeb-85ba-fdc28515aeff" containerName="istio-proxy"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338803    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="edf4f19d-74fd-4f83-96d2-6c66d711fa24" containerName="payload-pre-processing"
Oct 31 01:25:07.338773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338813    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c725e03-2ab7-49a9-8251-606dc8f70d4f" containerName="payload-pre-processing"
Oct 31 01:25:07.341509 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338820    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef7b8ccc-3c37-4ac0-bc1a-6b718d61c5bf" containerName="maas-api"
Oct 31 01:25:07.341509 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338830    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="da54fda6-be77-4967-8c98-8b1f99beeba6" containerName="istio-proxy"
Oct 31 01:25:07.341509 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338840    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="c853f186-bade-4124-b526-438799578a2c" containerName="payload-processing"
Oct 31 01:25:07.341509 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338847    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="39aa5706-7923-41cb-a27e-e2034f07e687" containerName="maas-api"
Oct 31 01:25:07.341509 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338855    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="a33d24c7-5486-4dba-aca9-83bedd6b6464" containerName="maas-api"
Oct 31 01:25:07.341509 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338863    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="2387e9d0-6540-45b2-b624-78f6f63e26f5" containerName="istio-proxy"
Oct 31 01:25:07.341509 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338871    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb16adc7-12a2-4497-a1a5-ad9adbe96684" containerName="payload-processing"
Oct 31 01:25:07.341509 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.338880    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5773d6f-3b5a-4e78-9d77-858c15e04639" containerName="manager"
Oct 31 01:25:07.342330 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.342309    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9c8d8b4d5-tfsdd"
Oct 31 01:25:07.358852 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.358832    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 31 01:25:07.371044 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.371020    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 31 01:25:07.371606 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.371584    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9c8d8b4d5-tfsdd"]
Oct 31 01:25:07.371754 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.371614    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 31 01:25:07.371754 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.371630    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jtxr8\""
Oct 31 01:25:07.498878 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.498846    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2073b6de-f1bb-410b-a250-34168362a697-webhook-cert\") pod \"maas-controller-9c8d8b4d5-tfsdd\" (UID: \"2073b6de-f1bb-410b-a250-34168362a697\") " pod="opendatahub/maas-controller-9c8d8b4d5-tfsdd"
Oct 31 01:25:07.499087 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.498901    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2073b6de-f1bb-410b-a250-34168362a697-metrics-certs\") pod \"maas-controller-9c8d8b4d5-tfsdd\" (UID: \"2073b6de-f1bb-410b-a250-34168362a697\") " pod="opendatahub/maas-controller-9c8d8b4d5-tfsdd"
Oct 31 01:25:07.499087 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.499044    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6thnz\" (UniqueName: \"kubernetes.io/projected/2073b6de-f1bb-410b-a250-34168362a697-kube-api-access-6thnz\") pod \"maas-controller-9c8d8b4d5-tfsdd\" (UID: \"2073b6de-f1bb-410b-a250-34168362a697\") " pod="opendatahub/maas-controller-9c8d8b4d5-tfsdd"
Oct 31 01:25:07.600329 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.600296    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2073b6de-f1bb-410b-a250-34168362a697-webhook-cert\") pod \"maas-controller-9c8d8b4d5-tfsdd\" (UID: \"2073b6de-f1bb-410b-a250-34168362a697\") " pod="opendatahub/maas-controller-9c8d8b4d5-tfsdd"
Oct 31 01:25:07.600512 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.600350    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2073b6de-f1bb-410b-a250-34168362a697-metrics-certs\") pod \"maas-controller-9c8d8b4d5-tfsdd\" (UID: \"2073b6de-f1bb-410b-a250-34168362a697\") " pod="opendatahub/maas-controller-9c8d8b4d5-tfsdd"
Oct 31 01:25:07.600512 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.600391    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6thnz\" (UniqueName: \"kubernetes.io/projected/2073b6de-f1bb-410b-a250-34168362a697-kube-api-access-6thnz\") pod \"maas-controller-9c8d8b4d5-tfsdd\" (UID: \"2073b6de-f1bb-410b-a250-34168362a697\") " pod="opendatahub/maas-controller-9c8d8b4d5-tfsdd"
Oct 31 01:25:07.602813 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.602786    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2073b6de-f1bb-410b-a250-34168362a697-webhook-cert\") pod \"maas-controller-9c8d8b4d5-tfsdd\" (UID: \"2073b6de-f1bb-410b-a250-34168362a697\") " pod="opendatahub/maas-controller-9c8d8b4d5-tfsdd"
Oct 31 01:25:07.602923 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.602831    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2073b6de-f1bb-410b-a250-34168362a697-metrics-certs\") pod \"maas-controller-9c8d8b4d5-tfsdd\" (UID: \"2073b6de-f1bb-410b-a250-34168362a697\") " pod="opendatahub/maas-controller-9c8d8b4d5-tfsdd"
Oct 31 01:25:07.624032 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.624004    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6thnz\" (UniqueName: \"kubernetes.io/projected/2073b6de-f1bb-410b-a250-34168362a697-kube-api-access-6thnz\") pod \"maas-controller-9c8d8b4d5-tfsdd\" (UID: \"2073b6de-f1bb-410b-a250-34168362a697\") " pod="opendatahub/maas-controller-9c8d8b4d5-tfsdd"
Oct 31 01:25:07.655012 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.654982    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9c8d8b4d5-tfsdd"
Oct 31 01:25:07.920555 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.920477    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5773d6f-3b5a-4e78-9d77-858c15e04639" path="/var/lib/kubelet/pods/b5773d6f-3b5a-4e78-9d77-858c15e04639/volumes"
Oct 31 01:25:07.921012 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.920996    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f32926f1-4427-4764-b86d-7294eaba9e0f" path="/var/lib/kubelet/pods/f32926f1-4427-4764-b86d-7294eaba9e0f/volumes"
Oct 31 01:25:07.952141 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.952116    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9c8d8b4d5-tfsdd"]
Oct 31 01:25:07.952805 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:25:07.952783    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2073b6de_f1bb_410b_a250_34168362a697.slice/crio-a808d2fc77d1763621f1bb552901513481eb5c8b46f71200d95a32aebe4c2cf0 WatchSource:0}: Error finding container a808d2fc77d1763621f1bb552901513481eb5c8b46f71200d95a32aebe4c2cf0: Status 404 returned error can't find the container with id a808d2fc77d1763621f1bb552901513481eb5c8b46f71200d95a32aebe4c2cf0
Oct 31 01:25:07.954100 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:07.954083    4327 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:25:08.304569 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:08.304486    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9c8d8b4d5-tfsdd" event={"ID":"2073b6de-f1bb-410b-a250-34168362a697","Type":"ContainerStarted","Data":"a808d2fc77d1763621f1bb552901513481eb5c8b46f71200d95a32aebe4c2cf0"}
Oct 31 01:25:09.311078 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:09.311043    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9c8d8b4d5-tfsdd" event={"ID":"2073b6de-f1bb-410b-a250-34168362a697","Type":"ContainerStarted","Data":"d44b4d324ccc94ef975a4834ba9a00695da278feed8eca59562bfbc78444893c"}
Oct 31 01:25:09.311604 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:09.311133    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9c8d8b4d5-tfsdd"
Oct 31 01:25:09.355367 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:09.355308    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-9c8d8b4d5-tfsdd" podStartSLOduration=1.891229106 podStartE2EDuration="2.355294058s" totalImagesPullingTime="464.064952ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:09.354536462 +0000 UTC m=+3969.576572003" watchObservedRunningTime="2026-10-31 01:25:09.355294058 +0000 UTC m=+3969.577329599"
Oct 31 01:25:15.291463 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:25:15.291422    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:25:20.318038 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:20.318007    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9c8d8b4d5-tfsdd"
Oct 31 01:25:35.643323 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.643287    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"]
Oct 31 01:25:35.646847 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.646828    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.677250 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.677213    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"]
Oct 31 01:25:35.759381 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.759344    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b1dad6d-c028-4673-96f1-48c4fca807c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.759572 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.759404    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9r6k\" (UniqueName: \"kubernetes.io/projected/9b1dad6d-c028-4673-96f1-48c4fca807c4-kube-api-access-k9r6k\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.759572 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.759513    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b1dad6d-c028-4673-96f1-48c4fca807c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.759572 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.759544    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b1dad6d-c028-4673-96f1-48c4fca807c4-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.759730 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.759586    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b1dad6d-c028-4673-96f1-48c4fca807c4-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.759730 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.759626    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b1dad6d-c028-4673-96f1-48c4fca807c4-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.759730 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.759654    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b1dad6d-c028-4673-96f1-48c4fca807c4-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.759839 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.759724    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b1dad6d-c028-4673-96f1-48c4fca807c4-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.759839 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.759750    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b1dad6d-c028-4673-96f1-48c4fca807c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.860653 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.860620    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9r6k\" (UniqueName: \"kubernetes.io/projected/9b1dad6d-c028-4673-96f1-48c4fca807c4-kube-api-access-k9r6k\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.860870 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.860661    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b1dad6d-c028-4673-96f1-48c4fca807c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.860870 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.860715    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b1dad6d-c028-4673-96f1-48c4fca807c4-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.860870 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.860763    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b1dad6d-c028-4673-96f1-48c4fca807c4-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.860870 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.860797    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b1dad6d-c028-4673-96f1-48c4fca807c4-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.860870 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.860828    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b1dad6d-c028-4673-96f1-48c4fca807c4-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.860870 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.860860    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b1dad6d-c028-4673-96f1-48c4fca807c4-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.861216 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.860884    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b1dad6d-c028-4673-96f1-48c4fca807c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.861216 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.860949    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b1dad6d-c028-4673-96f1-48c4fca807c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.861216 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.861182    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b1dad6d-c028-4673-96f1-48c4fca807c4-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.861363 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.861330    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b1dad6d-c028-4673-96f1-48c4fca807c4-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.861422 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.861377    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b1dad6d-c028-4673-96f1-48c4fca807c4-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.861483 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.861420    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b1dad6d-c028-4673-96f1-48c4fca807c4-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.861782 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.861760    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b1dad6d-c028-4673-96f1-48c4fca807c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.863231 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.863212    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b1dad6d-c028-4673-96f1-48c4fca807c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.863554 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.863533    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b1dad6d-c028-4673-96f1-48c4fca807c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.885052 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.885023    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b1dad6d-c028-4673-96f1-48c4fca807c4-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.885245 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.885222    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9r6k\" (UniqueName: \"kubernetes.io/projected/9b1dad6d-c028-4673-96f1-48c4fca807c4-kube-api-access-k9r6k\") pod \"maas-default-gateway-openshift-default-6886b96f9-zrjvf\" (UID: \"9b1dad6d-c028-4673-96f1-48c4fca807c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:35.960930 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:35.960839    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:36.270879 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:36.270849    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"]
Oct 31 01:25:36.271549 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:25:36.271525    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b1dad6d_c028_4673_96f1_48c4fca807c4.slice/crio-b0e45c2b0c943e9a3fe61fb04acb09ff77d8b928c496f83a94fa1c51723caf75 WatchSource:0}: Error finding container b0e45c2b0c943e9a3fe61fb04acb09ff77d8b928c496f83a94fa1c51723caf75: Status 404 returned error can't find the container with id b0e45c2b0c943e9a3fe61fb04acb09ff77d8b928c496f83a94fa1c51723caf75
Oct 31 01:25:36.273692 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:36.273642    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:25:36.273780 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:36.273738    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:25:36.273780 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:36.273769    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:25:36.430815 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:36.430779    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf" event={"ID":"9b1dad6d-c028-4673-96f1-48c4fca807c4","Type":"ContainerStarted","Data":"ad160870c6a416e60c732700d65e0c4b7a822407876aaf53a6272e672c857c23"}
Oct 31 01:25:36.430996 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:36.430835    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf" event={"ID":"9b1dad6d-c028-4673-96f1-48c4fca807c4","Type":"ContainerStarted","Data":"b0e45c2b0c943e9a3fe61fb04acb09ff77d8b928c496f83a94fa1c51723caf75"}
Oct 31 01:25:36.522975 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:36.522868    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf" podStartSLOduration=1.522853394 podStartE2EDuration="1.522853394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:36.52081134 +0000 UTC m=+3996.742846882" watchObservedRunningTime="2026-10-31 01:25:36.522853394 +0000 UTC m=+3996.744888936"
Oct 31 01:25:36.962102 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:36.961930    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:37.177666 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:37.177627    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9c8d8b4d5-tfsdd"]
Oct 31 01:25:37.177879 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:37.177848    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-9c8d8b4d5-tfsdd" podUID="2073b6de-f1bb-410b-a250-34168362a697" containerName="manager" containerID="cri-o://d44b4d324ccc94ef975a4834ba9a00695da278feed8eca59562bfbc78444893c" gracePeriod=30
Oct 31 01:25:37.450210 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:37.450178    4327 generic.go:356] "Generic (PLEG): container finished" podID="2073b6de-f1bb-410b-a250-34168362a697" containerID="d44b4d324ccc94ef975a4834ba9a00695da278feed8eca59562bfbc78444893c" exitCode=0
Oct 31 01:25:37.450369 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:37.450256    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9c8d8b4d5-tfsdd" event={"ID":"2073b6de-f1bb-410b-a250-34168362a697","Type":"ContainerDied","Data":"d44b4d324ccc94ef975a4834ba9a00695da278feed8eca59562bfbc78444893c"}
Oct 31 01:25:37.466367 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:37.466341    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9c8d8b4d5-tfsdd"
Oct 31 01:25:37.576210 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:37.576117    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2073b6de-f1bb-410b-a250-34168362a697-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2073b6de-f1bb-410b-a250-34168362a697-metrics-certs\") pod \"2073b6de-f1bb-410b-a250-34168362a697\" (UID: \"2073b6de-f1bb-410b-a250-34168362a697\") "
Oct 31 01:25:37.576210 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:37.576165    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2073b6de-f1bb-410b-a250-34168362a697-kube-api-access-6thnz\" (UniqueName: \"kubernetes.io/projected/2073b6de-f1bb-410b-a250-34168362a697-kube-api-access-6thnz\") pod \"2073b6de-f1bb-410b-a250-34168362a697\" (UID: \"2073b6de-f1bb-410b-a250-34168362a697\") "
Oct 31 01:25:37.576408 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:37.576225    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2073b6de-f1bb-410b-a250-34168362a697-webhook-cert\" (UniqueName: \"kubernetes.io/secret/2073b6de-f1bb-410b-a250-34168362a697-webhook-cert\") pod \"2073b6de-f1bb-410b-a250-34168362a697\" (UID: \"2073b6de-f1bb-410b-a250-34168362a697\") "
Oct 31 01:25:37.578569 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:37.578527    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2073b6de-f1bb-410b-a250-34168362a697-metrics-certs" pod "2073b6de-f1bb-410b-a250-34168362a697" (UID: "2073b6de-f1bb-410b-a250-34168362a697"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:25:37.578778 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:37.578562    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2073b6de-f1bb-410b-a250-34168362a697-webhook-cert" pod "2073b6de-f1bb-410b-a250-34168362a697" (UID: "2073b6de-f1bb-410b-a250-34168362a697"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:25:37.578778 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:37.578592    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2073b6de-f1bb-410b-a250-34168362a697-kube-api-access-6thnz" pod "2073b6de-f1bb-410b-a250-34168362a697" (UID: "2073b6de-f1bb-410b-a250-34168362a697"). InnerVolumeSpecName "kube-api-access-6thnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:25:37.677218 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:37.677165    4327 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2073b6de-f1bb-410b-a250-34168362a697-webhook-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:25:37.677218 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:37.677209    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2073b6de-f1bb-410b-a250-34168362a697-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:25:37.677218 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:37.677222    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6thnz\" (UniqueName: \"kubernetes.io/projected/2073b6de-f1bb-410b-a250-34168362a697-kube-api-access-6thnz\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:25:37.962572 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:37.962536    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zrjvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 31 01:25:37.962957 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:37.962598    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf" podUID="9b1dad6d-c028-4673-96f1-48c4fca807c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 31 01:25:38.455612 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:38.455575    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9c8d8b4d5-tfsdd"
Oct 31 01:25:38.455612 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:38.455598    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9c8d8b4d5-tfsdd" event={"ID":"2073b6de-f1bb-410b-a250-34168362a697","Type":"ContainerDied","Data":"a808d2fc77d1763621f1bb552901513481eb5c8b46f71200d95a32aebe4c2cf0"}
Oct 31 01:25:38.455892 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:38.455701    4327 scope.go:122] "RemoveContainer" containerID="d44b4d324ccc94ef975a4834ba9a00695da278feed8eca59562bfbc78444893c"
Oct 31 01:25:38.507546 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:38.507513    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9c8d8b4d5-tfsdd"]
Oct 31 01:25:38.540703 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:38.540462    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-9c8d8b4d5-tfsdd"]
Oct 31 01:25:38.961545 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:38.961507    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zrjvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 31 01:25:38.961768 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:38.961566    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf" podUID="9b1dad6d-c028-4673-96f1-48c4fca807c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 31 01:25:39.774220 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:39.774185    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9c8d8b4d5-z5sc9"]
Oct 31 01:25:39.774664 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:39.774566    4327 cpu_manager.go:414] "containerMap: removing container" podUID="2073b6de-f1bb-410b-a250-34168362a697" containerName="manager"
Oct 31 01:25:39.774664 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:39.774577    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2073b6de-f1bb-410b-a250-34168362a697" containerName="manager"
Oct 31 01:25:39.774664 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:39.774633    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="2073b6de-f1bb-410b-a250-34168362a697" containerName="manager"
Oct 31 01:25:39.778992 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:39.778972    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9c8d8b4d5-z5sc9"
Oct 31 01:25:39.794316 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:39.794294    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 31 01:25:39.806578 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:39.806550    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 31 01:25:39.806578 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:39.806554    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 31 01:25:39.806578 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:39.806562    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jtxr8\""
Oct 31 01:25:39.808608 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:39.808581    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9c8d8b4d5-z5sc9"]
Oct 31 01:25:39.899780 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:39.899737    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0121470e-c1cd-459c-ba8b-0c6f8cac9c4f-metrics-certs\") pod \"maas-controller-9c8d8b4d5-z5sc9\" (UID: \"0121470e-c1cd-459c-ba8b-0c6f8cac9c4f\") " pod="opendatahub/maas-controller-9c8d8b4d5-z5sc9"
Oct 31 01:25:39.899780 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:39.899779    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nms5r\" (UniqueName: \"kubernetes.io/projected/0121470e-c1cd-459c-ba8b-0c6f8cac9c4f-kube-api-access-nms5r\") pod \"maas-controller-9c8d8b4d5-z5sc9\" (UID: \"0121470e-c1cd-459c-ba8b-0c6f8cac9c4f\") " pod="opendatahub/maas-controller-9c8d8b4d5-z5sc9"
Oct 31 01:25:39.900013 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:39.899888    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0121470e-c1cd-459c-ba8b-0c6f8cac9c4f-webhook-cert\") pod \"maas-controller-9c8d8b4d5-z5sc9\" (UID: \"0121470e-c1cd-459c-ba8b-0c6f8cac9c4f\") " pod="opendatahub/maas-controller-9c8d8b4d5-z5sc9"
Oct 31 01:25:39.943944 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:39.943903    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2073b6de-f1bb-410b-a250-34168362a697" path="/var/lib/kubelet/pods/2073b6de-f1bb-410b-a250-34168362a697/volumes"
Oct 31 01:25:39.962316 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:39.962278    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zrjvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 31 01:25:39.962487 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:39.962356    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf" podUID="9b1dad6d-c028-4673-96f1-48c4fca807c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 31 01:25:40.001354 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:40.001303    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0121470e-c1cd-459c-ba8b-0c6f8cac9c4f-metrics-certs\") pod \"maas-controller-9c8d8b4d5-z5sc9\" (UID: \"0121470e-c1cd-459c-ba8b-0c6f8cac9c4f\") " pod="opendatahub/maas-controller-9c8d8b4d5-z5sc9"
Oct 31 01:25:40.001354 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:40.001359    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nms5r\" (UniqueName: \"kubernetes.io/projected/0121470e-c1cd-459c-ba8b-0c6f8cac9c4f-kube-api-access-nms5r\") pod \"maas-controller-9c8d8b4d5-z5sc9\" (UID: \"0121470e-c1cd-459c-ba8b-0c6f8cac9c4f\") " pod="opendatahub/maas-controller-9c8d8b4d5-z5sc9"
Oct 31 01:25:40.001610 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:40.001432    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0121470e-c1cd-459c-ba8b-0c6f8cac9c4f-webhook-cert\") pod \"maas-controller-9c8d8b4d5-z5sc9\" (UID: \"0121470e-c1cd-459c-ba8b-0c6f8cac9c4f\") " pod="opendatahub/maas-controller-9c8d8b4d5-z5sc9"
Oct 31 01:25:40.003809 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:40.003777    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0121470e-c1cd-459c-ba8b-0c6f8cac9c4f-metrics-certs\") pod \"maas-controller-9c8d8b4d5-z5sc9\" (UID: \"0121470e-c1cd-459c-ba8b-0c6f8cac9c4f\") " pod="opendatahub/maas-controller-9c8d8b4d5-z5sc9"
Oct 31 01:25:40.003935 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:40.003844    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0121470e-c1cd-459c-ba8b-0c6f8cac9c4f-webhook-cert\") pod \"maas-controller-9c8d8b4d5-z5sc9\" (UID: \"0121470e-c1cd-459c-ba8b-0c6f8cac9c4f\") " pod="opendatahub/maas-controller-9c8d8b4d5-z5sc9"
Oct 31 01:25:40.042741 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:40.039564    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nms5r\" (UniqueName: \"kubernetes.io/projected/0121470e-c1cd-459c-ba8b-0c6f8cac9c4f-kube-api-access-nms5r\") pod \"maas-controller-9c8d8b4d5-z5sc9\" (UID: \"0121470e-c1cd-459c-ba8b-0c6f8cac9c4f\") " pod="opendatahub/maas-controller-9c8d8b4d5-z5sc9"
Oct 31 01:25:40.090989 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:40.090955    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9c8d8b4d5-z5sc9"
Oct 31 01:25:40.409394 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:40.409360    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9c8d8b4d5-z5sc9"]
Oct 31 01:25:40.410213 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:25:40.410185    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0121470e_c1cd_459c_ba8b_0c6f8cac9c4f.slice/crio-8fb260be95025cb35dd0d95682067926c9b762603f3281227c41e3f88c7a28ec WatchSource:0}: Error finding container 8fb260be95025cb35dd0d95682067926c9b762603f3281227c41e3f88c7a28ec: Status 404 returned error can't find the container with id 8fb260be95025cb35dd0d95682067926c9b762603f3281227c41e3f88c7a28ec
Oct 31 01:25:40.466068 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:40.466031    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9c8d8b4d5-z5sc9" event={"ID":"0121470e-c1cd-459c-ba8b-0c6f8cac9c4f","Type":"ContainerStarted","Data":"8fb260be95025cb35dd0d95682067926c9b762603f3281227c41e3f88c7a28ec"}
Oct 31 01:25:40.961354 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:40.961320    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zrjvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 31 01:25:40.961815 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:40.961374    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf" podUID="9b1dad6d-c028-4673-96f1-48c4fca807c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 31 01:25:41.471637 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:41.471602    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9c8d8b4d5-z5sc9" event={"ID":"0121470e-c1cd-459c-ba8b-0c6f8cac9c4f","Type":"ContainerStarted","Data":"8459ddab757ce17f5426ca40eacb10156253a8e97f9bac0a0777fd71a2107cf4"}
Oct 31 01:25:41.471830 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:41.471704    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9c8d8b4d5-z5sc9"
Oct 31 01:25:41.540752 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:41.539800    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-9c8d8b4d5-z5sc9" podStartSLOduration=2.184449325 podStartE2EDuration="2.539780516s" totalImagesPullingTime="355.331191ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:41.537868578 +0000 UTC m=+4001.759904154" watchObservedRunningTime="2026-10-31 01:25:41.539780516 +0000 UTC m=+4001.761816063"
Oct 31 01:25:41.961875 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:41.961830    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zrjvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 31 01:25:41.962370 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:41.961892    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf" podUID="9b1dad6d-c028-4673-96f1-48c4fca807c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 31 01:25:42.961215 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:42.961178    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zrjvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 31 01:25:42.963288 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:42.961231    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf" podUID="9b1dad6d-c028-4673-96f1-48c4fca807c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 31 01:25:44.573511 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:44.572826    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:44.573511 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:44.573365    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:44.574764 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:44.574737    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zrjvf"
Oct 31 01:25:45.292200 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:25:45.292159    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:25:52.479162 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:25:52.479131    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9c8d8b4d5-z5sc9"
Oct 31 01:26:15.291974 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:26:15.291931    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:26:35.816106 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:35.816063    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"]
Oct 31 01:26:35.819984 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:35.819961    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:35.849645 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:35.849609    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"]
Oct 31 01:26:35.960822 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:35.960785    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97c56d5c-fa82-4ff9-b039-dd2839271bc7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:35.961042 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:35.960831    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97c56d5c-fa82-4ff9-b039-dd2839271bc7-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:35.961042 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:35.960865    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97c56d5c-fa82-4ff9-b039-dd2839271bc7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:35.961042 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:35.960952    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c56d5c-fa82-4ff9-b039-dd2839271bc7-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:35.961216 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:35.961049    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hcbt\" (UniqueName: \"kubernetes.io/projected/97c56d5c-fa82-4ff9-b039-dd2839271bc7-kube-api-access-5hcbt\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:35.961216 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:35.961142    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97c56d5c-fa82-4ff9-b039-dd2839271bc7-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:35.961216 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:35.961166    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c56d5c-fa82-4ff9-b039-dd2839271bc7-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:35.961216 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:35.961189    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97c56d5c-fa82-4ff9-b039-dd2839271bc7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:35.961361 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:35.961219    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97c56d5c-fa82-4ff9-b039-dd2839271bc7-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:36.062099 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:36.062065    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97c56d5c-fa82-4ff9-b039-dd2839271bc7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:36.062324 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:36.062117    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97c56d5c-fa82-4ff9-b039-dd2839271bc7-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:36.062324 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:36.062152    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97c56d5c-fa82-4ff9-b039-dd2839271bc7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:36.062324 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:36.062181    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c56d5c-fa82-4ff9-b039-dd2839271bc7-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:36.062324 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:36.062222    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hcbt\" (UniqueName: \"kubernetes.io/projected/97c56d5c-fa82-4ff9-b039-dd2839271bc7-kube-api-access-5hcbt\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:36.062324 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:36.062275    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97c56d5c-fa82-4ff9-b039-dd2839271bc7-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:36.062324 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:36.062303    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c56d5c-fa82-4ff9-b039-dd2839271bc7-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:36.062644 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:36.062334    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97c56d5c-fa82-4ff9-b039-dd2839271bc7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:36.062644 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:36.062372    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97c56d5c-fa82-4ff9-b039-dd2839271bc7-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:36.062787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:36.062699    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97c56d5c-fa82-4ff9-b039-dd2839271bc7-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:36.062787 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:36.062722    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c56d5c-fa82-4ff9-b039-dd2839271bc7-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:36.062892 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:36.062818    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c56d5c-fa82-4ff9-b039-dd2839271bc7-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:36.062949 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:36.062890    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97c56d5c-fa82-4ff9-b039-dd2839271bc7-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:36.063134 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:36.063107    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97c56d5c-fa82-4ff9-b039-dd2839271bc7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:36.064858 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:36.064828    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97c56d5c-fa82-4ff9-b039-dd2839271bc7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:36.064970 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:36.064886    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97c56d5c-fa82-4ff9-b039-dd2839271bc7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:36.087053 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:36.086999    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hcbt\" (UniqueName: \"kubernetes.io/projected/97c56d5c-fa82-4ff9-b039-dd2839271bc7-kube-api-access-5hcbt\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:36.087053 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:36.087017    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97c56d5c-fa82-4ff9-b039-dd2839271bc7-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-hlnsl\" (UID: \"97c56d5c-fa82-4ff9-b039-dd2839271bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:36.134618 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:36.134587    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:36.445746 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:36.445713    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"]
Oct 31 01:26:36.446219 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:26:36.446192    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c56d5c_fa82_4ff9_b039_dd2839271bc7.slice/crio-8017fdfdf9df59fc9482129856da3bfbe40d333eb2702ef7e30d8634e9e85aaa WatchSource:0}: Error finding container 8017fdfdf9df59fc9482129856da3bfbe40d333eb2702ef7e30d8634e9e85aaa: Status 404 returned error can't find the container with id 8017fdfdf9df59fc9482129856da3bfbe40d333eb2702ef7e30d8634e9e85aaa
Oct 31 01:26:36.448460 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:36.448428    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:26:36.448549 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:36.448495    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:26:36.448549 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:36.448525    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:26:36.744520 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:36.744422    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl" event={"ID":"97c56d5c-fa82-4ff9-b039-dd2839271bc7","Type":"ContainerStarted","Data":"eb048e2d179d00255b27130298563abce3046700ae49a368dd89e64d863fb5f6"}
Oct 31 01:26:36.744520 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:36.744486    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl" event={"ID":"97c56d5c-fa82-4ff9-b039-dd2839271bc7","Type":"ContainerStarted","Data":"8017fdfdf9df59fc9482129856da3bfbe40d333eb2702ef7e30d8634e9e85aaa"}
Oct 31 01:26:36.792136 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:36.792077    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl" podStartSLOduration=1.792062634 podStartE2EDuration="1.792062634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:26:36.790272392 +0000 UTC m=+4057.012307936" watchObservedRunningTime="2026-10-31 01:26:36.792062634 +0000 UTC m=+4057.014098176"
Oct 31 01:26:37.137702 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:37.137474    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:37.137702 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:37.137700    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hlnsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 31 01:26:37.138284 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:37.137755    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl" podUID="97c56d5c-fa82-4ff9-b039-dd2839271bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 31 01:26:38.138152 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:38.138110    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hlnsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 31 01:26:38.138837 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:38.138798    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl" podUID="97c56d5c-fa82-4ff9-b039-dd2839271bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 31 01:26:39.140704 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:39.137217    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hlnsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 31 01:26:39.140704 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:39.137296    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl" podUID="97c56d5c-fa82-4ff9-b039-dd2839271bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 31 01:26:40.138172 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:40.138133    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hlnsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 31 01:26:40.138469 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:40.138440    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl" podUID="97c56d5c-fa82-4ff9-b039-dd2839271bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 31 01:26:41.135179 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:41.135138    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hlnsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 31 01:26:41.136346 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:41.135904    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl" podUID="97c56d5c-fa82-4ff9-b039-dd2839271bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 31 01:26:42.141705 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:42.138777    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hlnsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 31 01:26:42.141705 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:42.138854    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl" podUID="97c56d5c-fa82-4ff9-b039-dd2839271bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 31 01:26:43.135630 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:43.135582    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hlnsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 31 01:26:43.135839 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:43.135657    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl" podUID="97c56d5c-fa82-4ff9-b039-dd2839271bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 31 01:26:45.140553 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:45.140500    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hlnsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:26:45.141238 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:45.141202    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl" podUID="97c56d5c-fa82-4ff9-b039-dd2839271bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:26:45.293382 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:26:45.293317    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:26:46.182884 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:46.182834    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hlnsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:26:46.183443 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:46.182903    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl" podUID="97c56d5c-fa82-4ff9-b039-dd2839271bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:26:47.184224 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:47.184178    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hlnsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:26:47.184819 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:47.184265    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl" podUID="97c56d5c-fa82-4ff9-b039-dd2839271bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:26:48.225912 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:48.225858    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hlnsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:26:48.226486 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:48.225939    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl" podUID="97c56d5c-fa82-4ff9-b039-dd2839271bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:26:49.119903 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:49.119869    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:49.120399 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:49.120376    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:26:49.121263 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:26:49.121242    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl"
Oct 31 01:27:15.291097 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:27:15.291007    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:27:20.201054 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:27:20.201003    4327 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hlnsl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:27:20.201940 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:27:20.201902    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hlnsl" podUID="97c56d5c-fa82-4ff9-b039-dd2839271bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:27:45.291638 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:27:45.291600    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:28:15.292123 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:28:15.292083    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:28:45.291981 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:28:45.291937    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:29:15.292142 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:29:15.292099    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:29:45.291826 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:29:45.291784    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:30:15.291335 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:30:15.291245    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:30:45.291828 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:30:45.291789    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:31:04.143873 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.143826    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"]
Oct 31 01:31:04.150104 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.150073    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.166090 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.166061    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-e8e612-openshift-default-dockercfg-8nz6j\""
Oct 31 01:31:04.180964 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.180933    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"]
Oct 31 01:31:04.182599 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.182566    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56f7a2a-9b64-4895-a075-fe1a715455a3-istiod-ca-cert\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.183136 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.182621    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-workload-certs\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.183136 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.182749    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-envoy\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.183136 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.182813    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-workload-socket\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.183136 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.182839    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-data\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.183136 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.182876    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-podinfo\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.183136 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.182972    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-token\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.183136 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.183129    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-credential-socket\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.183567 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.183189    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlk7d\" (UniqueName: \"kubernetes.io/projected/d56f7a2a-9b64-4895-a075-fe1a715455a3-kube-api-access-hlk7d\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.283777 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.283734    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-workload-socket\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.283777 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.283776    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-data\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.284068 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.283807    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-podinfo\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.284068 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.283835    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-token\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.284068 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.283894    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-credential-socket\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.284068 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.283930    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlk7d\" (UniqueName: \"kubernetes.io/projected/d56f7a2a-9b64-4895-a075-fe1a715455a3-kube-api-access-hlk7d\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.284068 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.284025    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56f7a2a-9b64-4895-a075-fe1a715455a3-istiod-ca-cert\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.284068 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.284063    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-workload-certs\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.284416 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.284087    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-envoy\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.284416 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.284283    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-data\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.284416 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.284379    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-credential-socket\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.284416 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.284400    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-workload-socket\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.284651 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.284627    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-workload-certs\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.284834 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.284815    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56f7a2a-9b64-4895-a075-fe1a715455a3-istiod-ca-cert\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.286326 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.286305    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-envoy\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.286418 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.286379    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-podinfo\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.309615 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.309584    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-token\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.309783 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.309709    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlk7d\" (UniqueName: \"kubernetes.io/projected/d56f7a2a-9b64-4895-a075-fe1a715455a3-kube-api-access-hlk7d\") pod \"e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") " pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.465843 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.465731    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:04.780113 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.780027    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"]
Oct 31 01:31:04.782467 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:31:04.782434    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56f7a2a_9b64_4895_a075_fe1a715455a3.slice/crio-d358f369e6da48ca8a425b6d63b6f336d968c47c0399a40e5acf10546f54da1c WatchSource:0}: Error finding container d358f369e6da48ca8a425b6d63b6f336d968c47c0399a40e5acf10546f54da1c: Status 404 returned error can't find the container with id d358f369e6da48ca8a425b6d63b6f336d968c47c0399a40e5acf10546f54da1c
Oct 31 01:31:04.784744 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.784698    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:31:04.784848 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.784777    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:31:04.784848 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:04.784817    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:31:05.156832 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:05.156793    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj" event={"ID":"d56f7a2a-9b64-4895-a075-fe1a715455a3","Type":"ContainerStarted","Data":"f9baaf0ee908497e71aea6fc3eab64b5ae3f54dc74b9c434ec3d4445e10dff49"}
Oct 31 01:31:05.157249 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:05.156854    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj" event={"ID":"d56f7a2a-9b64-4895-a075-fe1a715455a3","Type":"ContainerStarted","Data":"d358f369e6da48ca8a425b6d63b6f336d968c47c0399a40e5acf10546f54da1c"}
Oct 31 01:31:05.203295 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:05.203243    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj" podStartSLOduration=1.20322906 podStartE2EDuration="1.20322906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:31:05.202357437 +0000 UTC m=+4325.424392978" watchObservedRunningTime="2026-10-31 01:31:05.20322906 +0000 UTC m=+4325.425264602"
Oct 31 01:31:05.466379 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:05.466291    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:05.471499 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:05.471466    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:06.161550 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:06.161515    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:06.162465 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:06.162443    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:09.788893 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:09.788854    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k"]
Oct 31 01:31:09.792385 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:09.792367    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k"
Oct 31 01:31:09.825452 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:09.825426    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-e8e612\""
Oct 31 01:31:09.825600 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:09.825581    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-efac4dbc-dockercfg-j7qm8\""
Oct 31 01:31:09.827345 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:09.827328    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k"]
Oct 31 01:31:09.850063 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:09.850032    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrgrv\" (UniqueName: \"kubernetes.io/projected/b777786e-14f4-437f-b2e9-fe57461c1280-kube-api-access-lrgrv\") pod \"payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k\" (UID: \"b777786e-14f4-437f-b2e9-fe57461c1280\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k"
Oct 31 01:31:09.850227 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:09.850134    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b777786e-14f4-437f-b2e9-fe57461c1280-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k\" (UID: \"b777786e-14f4-437f-b2e9-fe57461c1280\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k"
Oct 31 01:31:09.891497 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:09.891450    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2"]
Oct 31 01:31:09.902644 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:09.902614    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2"
Oct 31 01:31:09.922185 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:09.922157    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2"]
Oct 31 01:31:09.950659 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:09.950626    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrgrv\" (UniqueName: \"kubernetes.io/projected/b777786e-14f4-437f-b2e9-fe57461c1280-kube-api-access-lrgrv\") pod \"payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k\" (UID: \"b777786e-14f4-437f-b2e9-fe57461c1280\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k"
Oct 31 01:31:09.950863 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:09.950734    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b777786e-14f4-437f-b2e9-fe57461c1280-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k\" (UID: \"b777786e-14f4-437f-b2e9-fe57461c1280\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k"
Oct 31 01:31:09.951485 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:09.951464    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b777786e-14f4-437f-b2e9-fe57461c1280-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k\" (UID: \"b777786e-14f4-437f-b2e9-fe57461c1280\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k"
Oct 31 01:31:09.974920 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:09.974887    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrgrv\" (UniqueName: \"kubernetes.io/projected/b777786e-14f4-437f-b2e9-fe57461c1280-kube-api-access-lrgrv\") pod \"payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k\" (UID: \"b777786e-14f4-437f-b2e9-fe57461c1280\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k"
Oct 31 01:31:10.052061 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:10.051969    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e930feb-b01a-4d51-9350-0819e66d30de-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2\" (UID: \"8e930feb-b01a-4d51-9350-0819e66d30de\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2"
Oct 31 01:31:10.052061 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:10.052028    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-646m9\" (UniqueName: \"kubernetes.io/projected/8e930feb-b01a-4d51-9350-0819e66d30de-kube-api-access-646m9\") pod \"payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2\" (UID: \"8e930feb-b01a-4d51-9350-0819e66d30de\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2"
Oct 31 01:31:10.105220 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:10.105165    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k"
Oct 31 01:31:10.158484 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:10.158433    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e930feb-b01a-4d51-9350-0819e66d30de-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2\" (UID: \"8e930feb-b01a-4d51-9350-0819e66d30de\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2"
Oct 31 01:31:10.158734 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:10.158535    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-646m9\" (UniqueName: \"kubernetes.io/projected/8e930feb-b01a-4d51-9350-0819e66d30de-kube-api-access-646m9\") pod \"payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2\" (UID: \"8e930feb-b01a-4d51-9350-0819e66d30de\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2"
Oct 31 01:31:10.159246 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:10.159209    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e930feb-b01a-4d51-9350-0819e66d30de-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2\" (UID: \"8e930feb-b01a-4d51-9350-0819e66d30de\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2"
Oct 31 01:31:10.186478 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:10.186442    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-646m9\" (UniqueName: \"kubernetes.io/projected/8e930feb-b01a-4d51-9350-0819e66d30de-kube-api-access-646m9\") pod \"payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2\" (UID: \"8e930feb-b01a-4d51-9350-0819e66d30de\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2"
Oct 31 01:31:10.219671 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:10.219632    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2"
Oct 31 01:31:10.446025 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:10.436002    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k"]
Oct 31 01:31:10.548657 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:10.548629    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2"]
Oct 31 01:31:10.551131 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:31:10.551103    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e930feb_b01a_4d51_9350_0819e66d30de.slice/crio-b331d86b55a0517690264d7e99aa9d04a341a145c0f6bef0d5983d8a9651550a WatchSource:0}: Error finding container b331d86b55a0517690264d7e99aa9d04a341a145c0f6bef0d5983d8a9651550a: Status 404 returned error can't find the container with id b331d86b55a0517690264d7e99aa9d04a341a145c0f6bef0d5983d8a9651550a
Oct 31 01:31:11.189294 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:11.189251    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k" event={"ID":"b777786e-14f4-437f-b2e9-fe57461c1280","Type":"ContainerStarted","Data":"01b3e71b99fee3225b454ddba8b2c70b2d71d7e6a8d8df6e64113d0765d14a14"}
Oct 31 01:31:11.189963 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:11.189936    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k"
Oct 31 01:31:11.190097 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:11.189977    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k" event={"ID":"b777786e-14f4-437f-b2e9-fe57461c1280","Type":"ContainerStarted","Data":"3b8ee3ed1af0b6bbe5783e7eb1cc726d6991d4eefbe021ced30ee8db5569d49e"}
Oct 31 01:31:11.191607 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:11.191583    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2" event={"ID":"8e930feb-b01a-4d51-9350-0819e66d30de","Type":"ContainerStarted","Data":"7c64c5a19a765e00a14657639ce2d33e9047186fd19264ae6a9b7bf00fbd77d9"}
Oct 31 01:31:11.191857 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:11.191787    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2" event={"ID":"8e930feb-b01a-4d51-9350-0819e66d30de","Type":"ContainerStarted","Data":"b331d86b55a0517690264d7e99aa9d04a341a145c0f6bef0d5983d8a9651550a"}
Oct 31 01:31:11.191857 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:11.191824    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2"
Oct 31 01:31:11.246754 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:11.246693    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k" podStartSLOduration=2.246661406 podStartE2EDuration="2.246661406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:31:11.244713012 +0000 UTC m=+4331.466748551" watchObservedRunningTime="2026-10-31 01:31:11.246661406 +0000 UTC m=+4331.468697013"
Oct 31 01:31:11.288083 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:11.288033    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2" podStartSLOduration=2.288014224 podStartE2EDuration="2.288014224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:31:11.287352577 +0000 UTC m=+4331.509388120" watchObservedRunningTime="2026-10-31 01:31:11.288014224 +0000 UTC m=+4331.510049767"
Oct 31 01:31:15.292078 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:31:15.292035    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:31:22.176911 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:22.175074    4327 patch_prober.go:29] interesting pod/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:31:22.176911 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:22.175158    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj" podUID="d56f7a2a-9b64-4895-a075-fe1a715455a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:31:22.200055 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:22.200030    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2"
Oct 31 01:31:22.256475 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:22.256445    4327 generic.go:356] "Generic (PLEG): container finished" podID="d56f7a2a-9b64-4895-a075-fe1a715455a3" containerID="f9baaf0ee908497e71aea6fc3eab64b5ae3f54dc74b9c434ec3d4445e10dff49" exitCode=0
Oct 31 01:31:22.256652 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:22.256490    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj" event={"ID":"d56f7a2a-9b64-4895-a075-fe1a715455a3","Type":"ContainerDied","Data":"f9baaf0ee908497e71aea6fc3eab64b5ae3f54dc74b9c434ec3d4445e10dff49"}
Oct 31 01:31:22.256875 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:22.256861    4327 scope.go:122] "RemoveContainer" containerID="f9baaf0ee908497e71aea6fc3eab64b5ae3f54dc74b9c434ec3d4445e10dff49"
Oct 31 01:31:22.258017 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:22.257985    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:31:22.258139 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:22.258095    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:31:22.258211 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:22.258164    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:31:23.203846 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:23.203809    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k"
Oct 31 01:31:23.269658 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:23.269623    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj" event={"ID":"d56f7a2a-9b64-4895-a075-fe1a715455a3","Type":"ContainerStarted","Data":"9292d8691daf43f9a1f9ebd84e46514d2d5829240ca5d9442a6baec8e9589cb3"}
Oct 31 01:31:23.466049 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:23.465957    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:23.466317 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:23.466160    4327 patch_prober.go:29] interesting pod/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 31 01:31:23.466511 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:23.466479    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj" podUID="d56f7a2a-9b64-4895-a075-fe1a715455a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 31 01:31:24.466577 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:24.466538    4327 patch_prober.go:29] interesting pod/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 31 01:31:24.467119 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:24.466600    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj" podUID="d56f7a2a-9b64-4895-a075-fe1a715455a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 31 01:31:25.466876 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:25.466842    4327 patch_prober.go:29] interesting pod/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 31 01:31:25.467305 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:25.466901    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj" podUID="d56f7a2a-9b64-4895-a075-fe1a715455a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 31 01:31:26.466563 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:26.466528    4327 patch_prober.go:29] interesting pod/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 31 01:31:26.466805 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:26.466580    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj" podUID="d56f7a2a-9b64-4895-a075-fe1a715455a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 31 01:31:27.466319 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:27.466282    4327 patch_prober.go:29] interesting pod/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 31 01:31:27.466755 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:27.466335    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj" podUID="d56f7a2a-9b64-4895-a075-fe1a715455a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 31 01:31:28.741090 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:28.741048    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:28.741625 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:28.741454    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:28.742619 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:31:28.742596    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:31:45.293751 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:31:45.293711    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:32:15.292325 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:32:15.292281    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:32:19.599496 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:19.599459    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2"]
Oct 31 01:32:19.632706 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:19.632658    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k"]
Oct 31 01:32:24.599805 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:24.599758    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2" podUID="8e930feb-b01a-4d51-9350-0819e66d30de" containerName="payload-processing" containerID="cri-o://7c64c5a19a765e00a14657639ce2d33e9047186fd19264ae6a9b7bf00fbd77d9" gracePeriod=25
Oct 31 01:32:24.633317 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:24.633270    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k" podUID="b777786e-14f4-437f-b2e9-fe57461c1280" containerName="payload-pre-processing" containerID="cri-o://01b3e71b99fee3225b454ddba8b2c70b2d71d7e6a8d8df6e64113d0765d14a14" gracePeriod=25
Oct 31 01:32:29.775528 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:29.775485    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"]
Oct 31 01:32:29.776294 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:29.776238    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj" podUID="d56f7a2a-9b64-4895-a075-fe1a715455a3" containerName="istio-proxy" containerID="cri-o://9292d8691daf43f9a1f9ebd84e46514d2d5829240ca5d9442a6baec8e9589cb3" gracePeriod=30
Oct 31 01:32:32.197489 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:32.197445    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2" podUID="8e930feb-b01a-4d51-9350-0819e66d30de" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:32:33.201702 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:33.201639    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k" podUID="b777786e-14f4-437f-b2e9-fe57461c1280" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:32:35.039084 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.039054    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:32:35.102829 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.102796    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-workload-socket\") pod \"d56f7a2a-9b64-4895-a075-fe1a715455a3\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") "
Oct 31 01:32:35.103000 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.102852    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-workload-certs\") pod \"d56f7a2a-9b64-4895-a075-fe1a715455a3\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") "
Oct 31 01:32:35.103000 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.102880    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-token\" (UniqueName: \"kubernetes.io/projected/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-token\") pod \"d56f7a2a-9b64-4895-a075-fe1a715455a3\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") "
Oct 31 01:32:35.103000 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.102958    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-credential-socket\") pod \"d56f7a2a-9b64-4895-a075-fe1a715455a3\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") "
Oct 31 01:32:35.103174 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.103015    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-envoy\") pod \"d56f7a2a-9b64-4895-a075-fe1a715455a3\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") "
Oct 31 01:32:35.103174 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.103063    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d56f7a2a-9b64-4895-a075-fe1a715455a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56f7a2a-9b64-4895-a075-fe1a715455a3-istiod-ca-cert\") pod \"d56f7a2a-9b64-4895-a075-fe1a715455a3\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") "
Oct 31 01:32:35.103174 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.103105    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-podinfo\") pod \"d56f7a2a-9b64-4895-a075-fe1a715455a3\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") "
Oct 31 01:32:35.103174 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.103084    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-workload-socket" pod "d56f7a2a-9b64-4895-a075-fe1a715455a3" (UID: "d56f7a2a-9b64-4895-a075-fe1a715455a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:32:35.103174 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.103164    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d56f7a2a-9b64-4895-a075-fe1a715455a3-kube-api-access-hlk7d\" (UniqueName: \"kubernetes.io/projected/d56f7a2a-9b64-4895-a075-fe1a715455a3-kube-api-access-hlk7d\") pod \"d56f7a2a-9b64-4895-a075-fe1a715455a3\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") "
Oct 31 01:32:35.103174 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.103145    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-workload-certs" pod "d56f7a2a-9b64-4895-a075-fe1a715455a3" (UID: "d56f7a2a-9b64-4895-a075-fe1a715455a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:32:35.103516 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.103214    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-data\") pod \"d56f7a2a-9b64-4895-a075-fe1a715455a3\" (UID: \"d56f7a2a-9b64-4895-a075-fe1a715455a3\") "
Oct 31 01:32:35.103581 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.103548    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:32:35.103581 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.103569    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:32:35.103706 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.103622    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-credential-socket" pod "d56f7a2a-9b64-4895-a075-fe1a715455a3" (UID: "d56f7a2a-9b64-4895-a075-fe1a715455a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:32:35.103768 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.103750    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-data" pod "d56f7a2a-9b64-4895-a075-fe1a715455a3" (UID: "d56f7a2a-9b64-4895-a075-fe1a715455a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:32:35.104323 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.104299    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d56f7a2a-9b64-4895-a075-fe1a715455a3-istiod-ca-cert" pod "d56f7a2a-9b64-4895-a075-fe1a715455a3" (UID: "d56f7a2a-9b64-4895-a075-fe1a715455a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:32:35.105817 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.105794    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-envoy" pod "d56f7a2a-9b64-4895-a075-fe1a715455a3" (UID: "d56f7a2a-9b64-4895-a075-fe1a715455a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:32:35.106424 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.106405    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-token" pod "d56f7a2a-9b64-4895-a075-fe1a715455a3" (UID: "d56f7a2a-9b64-4895-a075-fe1a715455a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:32:35.106823 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.106796    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-podinfo" pod "d56f7a2a-9b64-4895-a075-fe1a715455a3" (UID: "d56f7a2a-9b64-4895-a075-fe1a715455a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:32:35.106933 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.106836    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d56f7a2a-9b64-4895-a075-fe1a715455a3-kube-api-access-hlk7d" pod "d56f7a2a-9b64-4895-a075-fe1a715455a3" (UID: "d56f7a2a-9b64-4895-a075-fe1a715455a3"). InnerVolumeSpecName "kube-api-access-hlk7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:32:35.204725 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.204670    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:32:35.204725 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.204727    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:32:35.204923 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.204737    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56f7a2a-9b64-4895-a075-fe1a715455a3-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:32:35.204923 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.204746    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:32:35.204923 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.204755    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlk7d\" (UniqueName: \"kubernetes.io/projected/d56f7a2a-9b64-4895-a075-fe1a715455a3-kube-api-access-hlk7d\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:32:35.204923 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.204764    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:32:35.204923 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.204774    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d56f7a2a-9b64-4895-a075-fe1a715455a3-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:32:35.625727 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.625690    4327 generic.go:356] "Generic (PLEG): container finished" podID="d56f7a2a-9b64-4895-a075-fe1a715455a3" containerID="9292d8691daf43f9a1f9ebd84e46514d2d5829240ca5d9442a6baec8e9589cb3" exitCode=0
Oct 31 01:32:35.625921 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.625738    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"
Oct 31 01:32:35.625921 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.625780    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj" event={"ID":"d56f7a2a-9b64-4895-a075-fe1a715455a3","Type":"ContainerDied","Data":"9292d8691daf43f9a1f9ebd84e46514d2d5829240ca5d9442a6baec8e9589cb3"}
Oct 31 01:32:35.625921 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.625839    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj" event={"ID":"d56f7a2a-9b64-4895-a075-fe1a715455a3","Type":"ContainerDied","Data":"d358f369e6da48ca8a425b6d63b6f336d968c47c0399a40e5acf10546f54da1c"}
Oct 31 01:32:35.625921 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.625864    4327 scope.go:122] "RemoveContainer" containerID="9292d8691daf43f9a1f9ebd84e46514d2d5829240ca5d9442a6baec8e9589cb3"
Oct 31 01:32:35.635020 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.634986    4327 scope.go:122] "RemoveContainer" containerID="f9baaf0ee908497e71aea6fc3eab64b5ae3f54dc74b9c434ec3d4445e10dff49"
Oct 31 01:32:35.642905 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.642888    4327 scope.go:122] "RemoveContainer" containerID="9292d8691daf43f9a1f9ebd84e46514d2d5829240ca5d9442a6baec8e9589cb3"
Oct 31 01:32:35.643169 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:32:35.643152    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9292d8691daf43f9a1f9ebd84e46514d2d5829240ca5d9442a6baec8e9589cb3\": container with ID starting with 9292d8691daf43f9a1f9ebd84e46514d2d5829240ca5d9442a6baec8e9589cb3 not found: ID does not exist" containerID="9292d8691daf43f9a1f9ebd84e46514d2d5829240ca5d9442a6baec8e9589cb3"
Oct 31 01:32:35.643222 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.643177    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9292d8691daf43f9a1f9ebd84e46514d2d5829240ca5d9442a6baec8e9589cb3"} err="failed to get container status \"9292d8691daf43f9a1f9ebd84e46514d2d5829240ca5d9442a6baec8e9589cb3\": rpc error: code = NotFound desc = could not find container \"9292d8691daf43f9a1f9ebd84e46514d2d5829240ca5d9442a6baec8e9589cb3\": container with ID starting with 9292d8691daf43f9a1f9ebd84e46514d2d5829240ca5d9442a6baec8e9589cb3 not found: ID does not exist"
Oct 31 01:32:35.643222 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.643193    4327 scope.go:122] "RemoveContainer" containerID="f9baaf0ee908497e71aea6fc3eab64b5ae3f54dc74b9c434ec3d4445e10dff49"
Oct 31 01:32:35.643401 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:32:35.643384    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9baaf0ee908497e71aea6fc3eab64b5ae3f54dc74b9c434ec3d4445e10dff49\": container with ID starting with f9baaf0ee908497e71aea6fc3eab64b5ae3f54dc74b9c434ec3d4445e10dff49 not found: ID does not exist" containerID="f9baaf0ee908497e71aea6fc3eab64b5ae3f54dc74b9c434ec3d4445e10dff49"
Oct 31 01:32:35.643443 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.643408    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9baaf0ee908497e71aea6fc3eab64b5ae3f54dc74b9c434ec3d4445e10dff49"} err="failed to get container status \"f9baaf0ee908497e71aea6fc3eab64b5ae3f54dc74b9c434ec3d4445e10dff49\": rpc error: code = NotFound desc = could not find container \"f9baaf0ee908497e71aea6fc3eab64b5ae3f54dc74b9c434ec3d4445e10dff49\": container with ID starting with f9baaf0ee908497e71aea6fc3eab64b5ae3f54dc74b9c434ec3d4445e10dff49 not found: ID does not exist"
Oct 31 01:32:35.690773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.690738    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"]
Oct 31 01:32:35.702948 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.702924    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e8e612-openshift-default-585fb6ccd5xdkmj"]
Oct 31 01:32:35.919539 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:35.919460    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d56f7a2a-9b64-4895-a075-fe1a715455a3" path="/var/lib/kubelet/pods/d56f7a2a-9b64-4895-a075-fe1a715455a3/volumes"
Oct 31 01:32:42.197231 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:42.197194    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2" podUID="8e930feb-b01a-4d51-9350-0819e66d30de" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:32:43.201354 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:43.201312    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k" podUID="b777786e-14f4-437f-b2e9-fe57461c1280" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:32:45.205327 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.205211    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2"
Oct 31 01:32:45.213822 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.213802    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k"
Oct 31 01:32:45.251701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.250965    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e930feb-b01a-4d51-9350-0819e66d30de-kube-api-access-646m9\" (UniqueName: \"kubernetes.io/projected/8e930feb-b01a-4d51-9350-0819e66d30de-kube-api-access-646m9\") pod \"8e930feb-b01a-4d51-9350-0819e66d30de\" (UID: \"8e930feb-b01a-4d51-9350-0819e66d30de\") "
Oct 31 01:32:45.251701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.251089    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e930feb-b01a-4d51-9350-0819e66d30de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e930feb-b01a-4d51-9350-0819e66d30de-plugins-config-volume\") pod \"8e930feb-b01a-4d51-9350-0819e66d30de\" (UID: \"8e930feb-b01a-4d51-9350-0819e66d30de\") "
Oct 31 01:32:45.255699 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.252114    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e930feb-b01a-4d51-9350-0819e66d30de-plugins-config-volume" pod "8e930feb-b01a-4d51-9350-0819e66d30de" (UID: "8e930feb-b01a-4d51-9350-0819e66d30de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:32:45.294282 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.292869    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e930feb-b01a-4d51-9350-0819e66d30de-kube-api-access-646m9" pod "8e930feb-b01a-4d51-9350-0819e66d30de" (UID: "8e930feb-b01a-4d51-9350-0819e66d30de"). InnerVolumeSpecName "kube-api-access-646m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:32:45.354701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.354020    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b777786e-14f4-437f-b2e9-fe57461c1280-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b777786e-14f4-437f-b2e9-fe57461c1280-plugins-config-volume\") pod \"b777786e-14f4-437f-b2e9-fe57461c1280\" (UID: \"b777786e-14f4-437f-b2e9-fe57461c1280\") "
Oct 31 01:32:45.354701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.354189    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b777786e-14f4-437f-b2e9-fe57461c1280-kube-api-access-lrgrv\" (UniqueName: \"kubernetes.io/projected/b777786e-14f4-437f-b2e9-fe57461c1280-kube-api-access-lrgrv\") pod \"b777786e-14f4-437f-b2e9-fe57461c1280\" (UID: \"b777786e-14f4-437f-b2e9-fe57461c1280\") "
Oct 31 01:32:45.354701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.354500    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e930feb-b01a-4d51-9350-0819e66d30de-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:32:45.354701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.354524    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-646m9\" (UniqueName: \"kubernetes.io/projected/8e930feb-b01a-4d51-9350-0819e66d30de-kube-api-access-646m9\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:32:45.376159 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.376098    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b777786e-14f4-437f-b2e9-fe57461c1280-plugins-config-volume" pod "b777786e-14f4-437f-b2e9-fe57461c1280" (UID: "b777786e-14f4-437f-b2e9-fe57461c1280"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:32:45.397915 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.397881    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b777786e-14f4-437f-b2e9-fe57461c1280-kube-api-access-lrgrv" pod "b777786e-14f4-437f-b2e9-fe57461c1280" (UID: "b777786e-14f4-437f-b2e9-fe57461c1280"). InnerVolumeSpecName "kube-api-access-lrgrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:32:45.398875 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:32:45.398799    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:32:45.455892 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.455856    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrgrv\" (UniqueName: \"kubernetes.io/projected/b777786e-14f4-437f-b2e9-fe57461c1280-kube-api-access-lrgrv\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:32:45.456128 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.456110    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b777786e-14f4-437f-b2e9-fe57461c1280-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:32:45.694642 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.694608    4327 generic.go:356] "Generic (PLEG): container finished" podID="b777786e-14f4-437f-b2e9-fe57461c1280" containerID="01b3e71b99fee3225b454ddba8b2c70b2d71d7e6a8d8df6e64113d0765d14a14" exitCode=0
Oct 31 01:32:45.694847 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.694656    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k"
Oct 31 01:32:45.694847 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.694707    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k" event={"ID":"b777786e-14f4-437f-b2e9-fe57461c1280","Type":"ContainerDied","Data":"01b3e71b99fee3225b454ddba8b2c70b2d71d7e6a8d8df6e64113d0765d14a14"}
Oct 31 01:32:45.694847 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.694763    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k" event={"ID":"b777786e-14f4-437f-b2e9-fe57461c1280","Type":"ContainerDied","Data":"3b8ee3ed1af0b6bbe5783e7eb1cc726d6991d4eefbe021ced30ee8db5569d49e"}
Oct 31 01:32:45.694847 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.694791    4327 scope.go:122] "RemoveContainer" containerID="01b3e71b99fee3225b454ddba8b2c70b2d71d7e6a8d8df6e64113d0765d14a14"
Oct 31 01:32:45.696111 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.696074    4327 generic.go:356] "Generic (PLEG): container finished" podID="8e930feb-b01a-4d51-9350-0819e66d30de" containerID="7c64c5a19a765e00a14657639ce2d33e9047186fd19264ae6a9b7bf00fbd77d9" exitCode=0
Oct 31 01:32:45.696219 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.696114    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2" event={"ID":"8e930feb-b01a-4d51-9350-0819e66d30de","Type":"ContainerDied","Data":"7c64c5a19a765e00a14657639ce2d33e9047186fd19264ae6a9b7bf00fbd77d9"}
Oct 31 01:32:45.696219 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.696147    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2"
Oct 31 01:32:45.696219 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.696180    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2" event={"ID":"8e930feb-b01a-4d51-9350-0819e66d30de","Type":"ContainerDied","Data":"b331d86b55a0517690264d7e99aa9d04a341a145c0f6bef0d5983d8a9651550a"}
Oct 31 01:32:45.704777 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.704725    4327 scope.go:122] "RemoveContainer" containerID="01b3e71b99fee3225b454ddba8b2c70b2d71d7e6a8d8df6e64113d0765d14a14"
Oct 31 01:32:45.705019 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:32:45.704999    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01b3e71b99fee3225b454ddba8b2c70b2d71d7e6a8d8df6e64113d0765d14a14\": container with ID starting with 01b3e71b99fee3225b454ddba8b2c70b2d71d7e6a8d8df6e64113d0765d14a14 not found: ID does not exist" containerID="01b3e71b99fee3225b454ddba8b2c70b2d71d7e6a8d8df6e64113d0765d14a14"
Oct 31 01:32:45.705108 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.705031    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b3e71b99fee3225b454ddba8b2c70b2d71d7e6a8d8df6e64113d0765d14a14"} err="failed to get container status \"01b3e71b99fee3225b454ddba8b2c70b2d71d7e6a8d8df6e64113d0765d14a14\": rpc error: code = NotFound desc = could not find container \"01b3e71b99fee3225b454ddba8b2c70b2d71d7e6a8d8df6e64113d0765d14a14\": container with ID starting with 01b3e71b99fee3225b454ddba8b2c70b2d71d7e6a8d8df6e64113d0765d14a14 not found: ID does not exist"
Oct 31 01:32:45.705108 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.705054    4327 scope.go:122] "RemoveContainer" containerID="7c64c5a19a765e00a14657639ce2d33e9047186fd19264ae6a9b7bf00fbd77d9"
Oct 31 01:32:45.713028 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.713008    4327 scope.go:122] "RemoveContainer" containerID="7c64c5a19a765e00a14657639ce2d33e9047186fd19264ae6a9b7bf00fbd77d9"
Oct 31 01:32:45.713272 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:32:45.713255    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c64c5a19a765e00a14657639ce2d33e9047186fd19264ae6a9b7bf00fbd77d9\": container with ID starting with 7c64c5a19a765e00a14657639ce2d33e9047186fd19264ae6a9b7bf00fbd77d9 not found: ID does not exist" containerID="7c64c5a19a765e00a14657639ce2d33e9047186fd19264ae6a9b7bf00fbd77d9"
Oct 31 01:32:45.713336 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.713277    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c64c5a19a765e00a14657639ce2d33e9047186fd19264ae6a9b7bf00fbd77d9"} err="failed to get container status \"7c64c5a19a765e00a14657639ce2d33e9047186fd19264ae6a9b7bf00fbd77d9\": rpc error: code = NotFound desc = could not find container \"7c64c5a19a765e00a14657639ce2d33e9047186fd19264ae6a9b7bf00fbd77d9\": container with ID starting with 7c64c5a19a765e00a14657639ce2d33e9047186fd19264ae6a9b7bf00fbd77d9 not found: ID does not exist"
Oct 31 01:32:45.746101 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.746078    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2"]
Oct 31 01:32:45.749745 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.749728    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e8e612-59fc89c95x8pm2"]
Oct 31 01:32:45.795562 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.795528    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k"]
Oct 31 01:32:45.799963 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.799937    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8e612-77c78zv22k"]
Oct 31 01:32:45.919841 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.919770    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e930feb-b01a-4d51-9350-0819e66d30de" path="/var/lib/kubelet/pods/8e930feb-b01a-4d51-9350-0819e66d30de/volumes"
Oct 31 01:32:45.920177 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:32:45.920164    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b777786e-14f4-437f-b2e9-fe57461c1280" path="/var/lib/kubelet/pods/b777786e-14f4-437f-b2e9-fe57461c1280/volumes"
Oct 31 01:33:15.291185 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:33:15.291095    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:33:45.291447 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:33:45.291394    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:34:15.291088 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:34:15.291048    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:34:45.290708 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:34:45.290654    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:34:54.375789 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.375756    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"]
Oct 31 01:34:54.376293 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.376198    4327 cpu_manager.go:414] "containerMap: removing container" podUID="b777786e-14f4-437f-b2e9-fe57461c1280" containerName="payload-pre-processing"
Oct 31 01:34:54.376293 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.376214    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b777786e-14f4-437f-b2e9-fe57461c1280" containerName="payload-pre-processing"
Oct 31 01:34:54.376293 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.376244    4327 cpu_manager.go:414] "containerMap: removing container" podUID="d56f7a2a-9b64-4895-a075-fe1a715455a3" containerName="istio-proxy"
Oct 31 01:34:54.376293 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.376249    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d56f7a2a-9b64-4895-a075-fe1a715455a3" containerName="istio-proxy"
Oct 31 01:34:54.376293 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.376258    4327 cpu_manager.go:414] "containerMap: removing container" podUID="8e930feb-b01a-4d51-9350-0819e66d30de" containerName="payload-processing"
Oct 31 01:34:54.376293 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.376263    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e930feb-b01a-4d51-9350-0819e66d30de" containerName="payload-processing"
Oct 31 01:34:54.376293 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.376270    4327 cpu_manager.go:414] "containerMap: removing container" podUID="d56f7a2a-9b64-4895-a075-fe1a715455a3" containerName="istio-proxy"
Oct 31 01:34:54.376293 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.376274    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d56f7a2a-9b64-4895-a075-fe1a715455a3" containerName="istio-proxy"
Oct 31 01:34:54.376599 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.376333    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="d56f7a2a-9b64-4895-a075-fe1a715455a3" containerName="istio-proxy"
Oct 31 01:34:54.376599 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.376341    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e930feb-b01a-4d51-9350-0819e66d30de" containerName="payload-processing"
Oct 31 01:34:54.376599 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.376350    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="d56f7a2a-9b64-4895-a075-fe1a715455a3" containerName="istio-proxy"
Oct 31 01:34:54.376599 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.376358    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="b777786e-14f4-437f-b2e9-fe57461c1280" containerName="payload-pre-processing"
Oct 31 01:34:54.379805 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.379783    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.398194 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.398165    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-2e17bf-openshift-default-dockercfg-pdzjn\""
Oct 31 01:34:54.417384 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.417355    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"]
Oct 31 01:34:54.509713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.509289    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-credential-socket\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.509713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.509466    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-data\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.509713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.509565    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-envoy\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.509713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.509602    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-podinfo\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.509713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.509633    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istiod-ca-cert\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.509713 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.509700    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-token\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.510240 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.509745    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt6cv\" (UniqueName: \"kubernetes.io/projected/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-kube-api-access-gt6cv\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.510240 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.509828    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-workload-socket\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.510240 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.509869    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-workload-certs\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.610724 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.610663    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-credential-socket\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.611129 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.611094    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-data\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.611273 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.611173    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-envoy\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.611273 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.611224    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-podinfo\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.611273 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.611258    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istiod-ca-cert\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.611458 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.611308    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-token\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.611458 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.611354    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt6cv\" (UniqueName: \"kubernetes.io/projected/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-kube-api-access-gt6cv\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.611458 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.611417    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-credential-socket\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.611696 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.611428    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-workload-socket\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.611791 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.611746    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-workload-certs\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.611791 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.611762    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-data\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.612767 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.612550    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istiod-ca-cert\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.613236 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.612623    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-workload-socket\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.613377 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.613183    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-workload-certs\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.615415 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.615388    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-podinfo\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.615514 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.615497    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-envoy\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.639412 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.639352    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-token\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.639531 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.639451    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt6cv\" (UniqueName: \"kubernetes.io/projected/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-kube-api-access-gt6cv\") pod \"e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") " pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:54.693652 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:54.693625    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:55.008669 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:55.008643    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"]
Oct 31 01:34:55.011065 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:34:55.011034    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e929816_7500_4cc0_b2a4_eb50cb2f6be7.slice/crio-557d3c0a2596dae3cc5f5494621e52a9d997e888feda6188c861cffdd1b2a587 WatchSource:0}: Error finding container 557d3c0a2596dae3cc5f5494621e52a9d997e888feda6188c861cffdd1b2a587: Status 404 returned error can't find the container with id 557d3c0a2596dae3cc5f5494621e52a9d997e888feda6188c861cffdd1b2a587
Oct 31 01:34:55.013628 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:55.013594    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:34:55.013711 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:55.013662    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:34:55.013756 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:55.013716    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:34:55.271943 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:55.271838    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv" event={"ID":"8e929816-7500-4cc0-b2a4-eb50cb2f6be7","Type":"ContainerStarted","Data":"4c873f819e3749e4d0a6cbf0b2ef1d2a2bcc9ac91a056c4350383e58bc78cadc"}
Oct 31 01:34:55.271943 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:55.271899    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv" event={"ID":"8e929816-7500-4cc0-b2a4-eb50cb2f6be7","Type":"ContainerStarted","Data":"557d3c0a2596dae3cc5f5494621e52a9d997e888feda6188c861cffdd1b2a587"}
Oct 31 01:34:55.320160 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:55.320091    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv" podStartSLOduration=1.320073495 podStartE2EDuration="1.320073495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:34:55.317938028 +0000 UTC m=+4555.539973571" watchObservedRunningTime="2026-10-31 01:34:55.320073495 +0000 UTC m=+4555.542109038"
Oct 31 01:34:55.694457 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:55.694421    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:56.701024 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:56.700992    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:57.280826 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:57.280791    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:57.281695 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:57.281657    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:34:59.741012 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:59.740964    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg"]
Oct 31 01:34:59.744810 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:59.744780    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg"
Oct 31 01:34:59.761259 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:59.761220    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-2e17bf\""
Oct 31 01:34:59.773895 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:59.773859    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-677a7124-dockercfg-hcncp\""
Oct 31 01:34:59.776573 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:59.776544    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg"]
Oct 31 01:34:59.814443 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:59.814415    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc"]
Oct 31 01:34:59.818057 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:59.818031    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc"
Oct 31 01:34:59.858154 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:59.846871    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc"]
Oct 31 01:34:59.872607 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:59.872490    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcg96\" (UniqueName: \"kubernetes.io/projected/ce82c784-9eb5-43cb-9f23-d971046fbaf7-kube-api-access-hcg96\") pod \"payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg\" (UID: \"ce82c784-9eb5-43cb-9f23-d971046fbaf7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg"
Oct 31 01:34:59.873029 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:59.872994    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce82c784-9eb5-43cb-9f23-d971046fbaf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg\" (UID: \"ce82c784-9eb5-43cb-9f23-d971046fbaf7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg"
Oct 31 01:34:59.974264 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:59.974231    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcg96\" (UniqueName: \"kubernetes.io/projected/ce82c784-9eb5-43cb-9f23-d971046fbaf7-kube-api-access-hcg96\") pod \"payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg\" (UID: \"ce82c784-9eb5-43cb-9f23-d971046fbaf7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg"
Oct 31 01:34:59.974458 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:59.974321    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25a47e57-740b-46e3-af41-65ad3f8d7057-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc\" (UID: \"25a47e57-740b-46e3-af41-65ad3f8d7057\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc"
Oct 31 01:34:59.974458 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:59.974353    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxz44\" (UniqueName: \"kubernetes.io/projected/25a47e57-740b-46e3-af41-65ad3f8d7057-kube-api-access-gxz44\") pod \"payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc\" (UID: \"25a47e57-740b-46e3-af41-65ad3f8d7057\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc"
Oct 31 01:34:59.974458 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:59.974387    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce82c784-9eb5-43cb-9f23-d971046fbaf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg\" (UID: \"ce82c784-9eb5-43cb-9f23-d971046fbaf7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg"
Oct 31 01:34:59.975034 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:59.975014    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce82c784-9eb5-43cb-9f23-d971046fbaf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg\" (UID: \"ce82c784-9eb5-43cb-9f23-d971046fbaf7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg"
Oct 31 01:34:59.997943 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:34:59.997886    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcg96\" (UniqueName: \"kubernetes.io/projected/ce82c784-9eb5-43cb-9f23-d971046fbaf7-kube-api-access-hcg96\") pod \"payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg\" (UID: \"ce82c784-9eb5-43cb-9f23-d971046fbaf7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg"
Oct 31 01:35:00.072092 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:00.072054    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-677a7124-dockercfg-hcncp\""
Oct 31 01:35:00.075420 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:00.075394    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25a47e57-740b-46e3-af41-65ad3f8d7057-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc\" (UID: \"25a47e57-740b-46e3-af41-65ad3f8d7057\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc"
Oct 31 01:35:00.075595 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:00.075435    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxz44\" (UniqueName: \"kubernetes.io/projected/25a47e57-740b-46e3-af41-65ad3f8d7057-kube-api-access-gxz44\") pod \"payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc\" (UID: \"25a47e57-740b-46e3-af41-65ad3f8d7057\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc"
Oct 31 01:35:00.076145 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:00.076122    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25a47e57-740b-46e3-af41-65ad3f8d7057-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc\" (UID: \"25a47e57-740b-46e3-af41-65ad3f8d7057\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc"
Oct 31 01:35:00.079212 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:00.079185    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg"
Oct 31 01:35:00.100268 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:00.100239    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxz44\" (UniqueName: \"kubernetes.io/projected/25a47e57-740b-46e3-af41-65ad3f8d7057-kube-api-access-gxz44\") pod \"payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc\" (UID: \"25a47e57-740b-46e3-af41-65ad3f8d7057\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc"
Oct 31 01:35:00.173533 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:00.173067    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc"
Oct 31 01:35:00.397963 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:00.397936    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg"]
Oct 31 01:35:00.399764 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:35:00.399728    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce82c784_9eb5_43cb_9f23_d971046fbaf7.slice/crio-4c675620e748c6420c2444656b0aab8fbb92acc73b182f1789dd510a7ca38368 WatchSource:0}: Error finding container 4c675620e748c6420c2444656b0aab8fbb92acc73b182f1789dd510a7ca38368: Status 404 returned error can't find the container with id 4c675620e748c6420c2444656b0aab8fbb92acc73b182f1789dd510a7ca38368
Oct 31 01:35:00.473277 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:00.473243    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc"]
Oct 31 01:35:00.484980 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:35:00.484826    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a47e57_740b_46e3_af41_65ad3f8d7057.slice/crio-f3eca892832a5a8dee8892ff9fa14f1ebd924bbcbd3ca78d9e12ea8cf1407b19 WatchSource:0}: Error finding container f3eca892832a5a8dee8892ff9fa14f1ebd924bbcbd3ca78d9e12ea8cf1407b19: Status 404 returned error can't find the container with id f3eca892832a5a8dee8892ff9fa14f1ebd924bbcbd3ca78d9e12ea8cf1407b19
Oct 31 01:35:01.302525 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:01.302492    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg" event={"ID":"ce82c784-9eb5-43cb-9f23-d971046fbaf7","Type":"ContainerStarted","Data":"c823067778c221557e1dc79de9d881bc2d75e73e5887020198bd6e5605b19e4b"}
Oct 31 01:35:01.303043 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:01.302550    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg" event={"ID":"ce82c784-9eb5-43cb-9f23-d971046fbaf7","Type":"ContainerStarted","Data":"4c675620e748c6420c2444656b0aab8fbb92acc73b182f1789dd510a7ca38368"}
Oct 31 01:35:01.303043 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:01.302661    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg"
Oct 31 01:35:01.304183 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:01.304154    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc" event={"ID":"25a47e57-740b-46e3-af41-65ad3f8d7057","Type":"ContainerStarted","Data":"d0ad806c0f99eacbc1c36305392cba999330a7712a0557e252c530b142b71fc1"}
Oct 31 01:35:01.304267 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:01.304208    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc" event={"ID":"25a47e57-740b-46e3-af41-65ad3f8d7057","Type":"ContainerStarted","Data":"f3eca892832a5a8dee8892ff9fa14f1ebd924bbcbd3ca78d9e12ea8cf1407b19"}
Oct 31 01:35:01.304267 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:01.304237    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc"
Oct 31 01:35:01.349362 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:01.349310    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg" podStartSLOduration=2.349297026 podStartE2EDuration="2.349297026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:35:01.348137974 +0000 UTC m=+4561.570173517" watchObservedRunningTime="2026-10-31 01:35:01.349297026 +0000 UTC m=+4561.571332565"
Oct 31 01:35:01.389992 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:01.389932    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc" podStartSLOduration=2.389916775 podStartE2EDuration="2.389916775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:35:01.389399968 +0000 UTC m=+4561.611435511" watchObservedRunningTime="2026-10-31 01:35:01.389916775 +0000 UTC m=+4561.611952318"
Oct 31 01:35:12.311485 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:12.311455    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg"
Oct 31 01:35:12.311905 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:12.311530    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc"
Oct 31 01:35:13.224313 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:13.223216    4327 patch_prober.go:29] interesting pod/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": read tcp 10.133.0.2:56866->10.133.0.255:15021: read: connection reset by peer" start-of-body=
Oct 31 01:35:13.224313 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:13.223290    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv" podUID="8e929816-7500-4cc0-b2a4-eb50cb2f6be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": read tcp 10.133.0.2:56866->10.133.0.255:15021: read: connection reset by peer"
Oct 31 01:35:13.383329 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:13.383252    4327 generic.go:356] "Generic (PLEG): container finished" podID="8e929816-7500-4cc0-b2a4-eb50cb2f6be7" containerID="4c873f819e3749e4d0a6cbf0b2ef1d2a2bcc9ac91a056c4350383e58bc78cadc" exitCode=0
Oct 31 01:35:13.383720 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:13.383320    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv" event={"ID":"8e929816-7500-4cc0-b2a4-eb50cb2f6be7","Type":"ContainerDied","Data":"4c873f819e3749e4d0a6cbf0b2ef1d2a2bcc9ac91a056c4350383e58bc78cadc"}
Oct 31 01:35:13.383772 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:13.383749    4327 scope.go:122] "RemoveContainer" containerID="4c873f819e3749e4d0a6cbf0b2ef1d2a2bcc9ac91a056c4350383e58bc78cadc"
Oct 31 01:35:13.384948 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:13.384922    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:35:13.385042 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:13.385028    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:35:13.385082 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:13.385066    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:35:14.390023 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:14.389989    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv" event={"ID":"8e929816-7500-4cc0-b2a4-eb50cb2f6be7","Type":"ContainerStarted","Data":"9ab45c30b000205c4c40c9450149a00fa89f46c50a74a9a7bd8fe4d0d56771dc"}
Oct 31 01:35:14.694227 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:14.694130    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:35:14.694461 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:14.694434    4327 patch_prober.go:29] interesting pod/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 31 01:35:14.694552 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:14.694497    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv" podUID="8e929816-7500-4cc0-b2a4-eb50cb2f6be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 31 01:35:15.296371 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:35:15.296335    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:35:15.694956 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:15.694919    4327 patch_prober.go:29] interesting pod/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 31 01:35:15.695365 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:15.694976    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv" podUID="8e929816-7500-4cc0-b2a4-eb50cb2f6be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 31 01:35:16.694253 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:16.694214    4327 patch_prober.go:29] interesting pod/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 31 01:35:16.694432 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:16.694281    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv" podUID="8e929816-7500-4cc0-b2a4-eb50cb2f6be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 31 01:35:17.694548 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:17.694513    4327 patch_prober.go:29] interesting pod/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 31 01:35:17.694973 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:17.694564    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv" podUID="8e929816-7500-4cc0-b2a4-eb50cb2f6be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 31 01:35:18.694745 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:18.694713    4327 patch_prober.go:29] interesting pod/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 31 01:35:18.695152 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:18.694767    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv" podUID="8e929816-7500-4cc0-b2a4-eb50cb2f6be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 31 01:35:19.697933 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:19.697899    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:35:19.698333 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:19.698157    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:35:19.698798 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:35:19.698777    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:35:45.291521 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:35:45.291474    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:36:07.336227 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:07.336188    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc"]
Oct 31 01:36:07.365102 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:07.365071    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg"]
Oct 31 01:36:12.336773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:12.336659    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc" podUID="25a47e57-740b-46e3-af41-65ad3f8d7057" containerName="payload-processing" containerID="cri-o://d0ad806c0f99eacbc1c36305392cba999330a7712a0557e252c530b142b71fc1" gracePeriod=25
Oct 31 01:36:12.366303 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:12.366270    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg" podUID="ce82c784-9eb5-43cb-9f23-d971046fbaf7" containerName="payload-pre-processing" containerID="cri-o://c823067778c221557e1dc79de9d881bc2d75e73e5887020198bd6e5605b19e4b" gracePeriod=25
Oct 31 01:36:15.291127 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:36:15.291090    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:36:15.579101 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:15.579018    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"]
Oct 31 01:36:15.579329 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:15.579301    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv" podUID="8e929816-7500-4cc0-b2a4-eb50cb2f6be7" containerName="istio-proxy" containerID="cri-o://9ab45c30b000205c4c40c9450149a00fa89f46c50a74a9a7bd8fe4d0d56771dc" gracePeriod=30
Oct 31 01:36:19.700804 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:19.700771    4327 patch_prober.go:29] interesting pod/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:36:19.701264 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:19.700827    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv" podUID="8e929816-7500-4cc0-b2a4-eb50cb2f6be7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:36:20.718049 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:20.718018    4327 generic.go:356] "Generic (PLEG): container finished" podID="8e929816-7500-4cc0-b2a4-eb50cb2f6be7" containerID="9ab45c30b000205c4c40c9450149a00fa89f46c50a74a9a7bd8fe4d0d56771dc" exitCode=0
Oct 31 01:36:20.718440 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:20.718083    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv" event={"ID":"8e929816-7500-4cc0-b2a4-eb50cb2f6be7","Type":"ContainerDied","Data":"9ab45c30b000205c4c40c9450149a00fa89f46c50a74a9a7bd8fe4d0d56771dc"}
Oct 31 01:36:20.718440 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:20.718147    4327 scope.go:122] "RemoveContainer" containerID="4c873f819e3749e4d0a6cbf0b2ef1d2a2bcc9ac91a056c4350383e58bc78cadc"
Oct 31 01:36:20.839413 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:20.839388    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:36:21.015589 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.015551    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-data\") pod \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") "
Oct 31 01:36:21.015778 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.015617    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-envoy\") pod \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") "
Oct 31 01:36:21.015778 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.015668    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-credential-socket\") pod \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") "
Oct 31 01:36:21.015778 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.015727    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-workload-certs\") pod \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") "
Oct 31 01:36:21.015778 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.015753    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-workload-socket\") pod \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") "
Oct 31 01:36:21.015997 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.015783    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-kube-api-access-gt6cv\" (UniqueName: \"kubernetes.io/projected/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-kube-api-access-gt6cv\") pod \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") "
Oct 31 01:36:21.015997 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.015791    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-data" pod "8e929816-7500-4cc0-b2a4-eb50cb2f6be7" (UID: "8e929816-7500-4cc0-b2a4-eb50cb2f6be7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:21.015997 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.015823    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-token\" (UniqueName: \"kubernetes.io/projected/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-token\") pod \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") "
Oct 31 01:36:21.015997 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.015871    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-podinfo\") pod \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") "
Oct 31 01:36:21.015997 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.015917    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istiod-ca-cert\") pod \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\" (UID: \"8e929816-7500-4cc0-b2a4-eb50cb2f6be7\") "
Oct 31 01:36:21.016244 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.016009    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-workload-socket" pod "8e929816-7500-4cc0-b2a4-eb50cb2f6be7" (UID: "8e929816-7500-4cc0-b2a4-eb50cb2f6be7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:21.016244 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.016019    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-workload-certs" pod "8e929816-7500-4cc0-b2a4-eb50cb2f6be7" (UID: "8e929816-7500-4cc0-b2a4-eb50cb2f6be7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:21.016361 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.016328    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:36:21.016361 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.016349    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:36:21.016480 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.016363    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:36:21.016653 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.016614    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-credential-socket" pod "8e929816-7500-4cc0-b2a4-eb50cb2f6be7" (UID: "8e929816-7500-4cc0-b2a4-eb50cb2f6be7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:21.016821 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.016661    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istiod-ca-cert" pod "8e929816-7500-4cc0-b2a4-eb50cb2f6be7" (UID: "8e929816-7500-4cc0-b2a4-eb50cb2f6be7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:36:21.017969 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.017950    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-envoy" pod "8e929816-7500-4cc0-b2a4-eb50cb2f6be7" (UID: "8e929816-7500-4cc0-b2a4-eb50cb2f6be7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:21.018459 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.018438    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-podinfo" pod "8e929816-7500-4cc0-b2a4-eb50cb2f6be7" (UID: "8e929816-7500-4cc0-b2a4-eb50cb2f6be7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:36:21.018537 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.018478    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-kube-api-access-gt6cv" pod "8e929816-7500-4cc0-b2a4-eb50cb2f6be7" (UID: "8e929816-7500-4cc0-b2a4-eb50cb2f6be7"). InnerVolumeSpecName "kube-api-access-gt6cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:21.018583 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.018530    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-token" pod "8e929816-7500-4cc0-b2a4-eb50cb2f6be7" (UID: "8e929816-7500-4cc0-b2a4-eb50cb2f6be7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:21.117795 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.117764    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:36:21.117795 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.117789    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:36:21.117795 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.117800    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt6cv\" (UniqueName: \"kubernetes.io/projected/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-kube-api-access-gt6cv\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:36:21.118034 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.117809    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:36:21.118034 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.117818    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:36:21.118034 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.117826    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e929816-7500-4cc0-b2a4-eb50cb2f6be7-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:36:21.724097 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.724067    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"
Oct 31 01:36:21.724524 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.724074    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv" event={"ID":"8e929816-7500-4cc0-b2a4-eb50cb2f6be7","Type":"ContainerDied","Data":"557d3c0a2596dae3cc5f5494621e52a9d997e888feda6188c861cffdd1b2a587"}
Oct 31 01:36:21.724524 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.724191    4327 scope.go:122] "RemoveContainer" containerID="9ab45c30b000205c4c40c9450149a00fa89f46c50a74a9a7bd8fe4d0d56771dc"
Oct 31 01:36:21.788459 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.788423    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"]
Oct 31 01:36:21.800979 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.800948    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2e17bf-openshift-default-9f6bcd5d5-rcgnv"]
Oct 31 01:36:21.920109 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:21.920079    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e929816-7500-4cc0-b2a4-eb50cb2f6be7" path="/var/lib/kubelet/pods/8e929816-7500-4cc0-b2a4-eb50cb2f6be7/volumes"
Oct 31 01:36:22.310311 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:22.310269    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg" podUID="ce82c784-9eb5-43cb-9f23-d971046fbaf7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:36:22.310311 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:22.310286    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc" podUID="25a47e57-740b-46e3-af41-65ad3f8d7057" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:36:32.310371 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.310328    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc" podUID="25a47e57-740b-46e3-af41-65ad3f8d7057" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:36:32.310799 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.310370    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg" podUID="ce82c784-9eb5-43cb-9f23-d971046fbaf7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:36:32.615664 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.615635    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc"
Oct 31 01:36:32.639852 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.639820    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg"
Oct 31 01:36:32.642265 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.642229    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25a47e57-740b-46e3-af41-65ad3f8d7057-kube-api-access-gxz44\" (UniqueName: \"kubernetes.io/projected/25a47e57-740b-46e3-af41-65ad3f8d7057-kube-api-access-gxz44\") pod \"25a47e57-740b-46e3-af41-65ad3f8d7057\" (UID: \"25a47e57-740b-46e3-af41-65ad3f8d7057\") "
Oct 31 01:36:32.642402 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.642294    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25a47e57-740b-46e3-af41-65ad3f8d7057-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25a47e57-740b-46e3-af41-65ad3f8d7057-plugins-config-volume\") pod \"25a47e57-740b-46e3-af41-65ad3f8d7057\" (UID: \"25a47e57-740b-46e3-af41-65ad3f8d7057\") "
Oct 31 01:36:32.642693 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.642654    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25a47e57-740b-46e3-af41-65ad3f8d7057-plugins-config-volume" pod "25a47e57-740b-46e3-af41-65ad3f8d7057" (UID: "25a47e57-740b-46e3-af41-65ad3f8d7057"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:36:32.645142 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.645120    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a47e57-740b-46e3-af41-65ad3f8d7057-kube-api-access-gxz44" pod "25a47e57-740b-46e3-af41-65ad3f8d7057" (UID: "25a47e57-740b-46e3-af41-65ad3f8d7057"). InnerVolumeSpecName "kube-api-access-gxz44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:32.743555 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.743520    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce82c784-9eb5-43cb-9f23-d971046fbaf7-kube-api-access-hcg96\" (UniqueName: \"kubernetes.io/projected/ce82c784-9eb5-43cb-9f23-d971046fbaf7-kube-api-access-hcg96\") pod \"ce82c784-9eb5-43cb-9f23-d971046fbaf7\" (UID: \"ce82c784-9eb5-43cb-9f23-d971046fbaf7\") "
Oct 31 01:36:32.743767 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.743579    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce82c784-9eb5-43cb-9f23-d971046fbaf7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce82c784-9eb5-43cb-9f23-d971046fbaf7-plugins-config-volume\") pod \"ce82c784-9eb5-43cb-9f23-d971046fbaf7\" (UID: \"ce82c784-9eb5-43cb-9f23-d971046fbaf7\") "
Oct 31 01:36:32.743857 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.743779    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxz44\" (UniqueName: \"kubernetes.io/projected/25a47e57-740b-46e3-af41-65ad3f8d7057-kube-api-access-gxz44\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:36:32.743857 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.743793    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25a47e57-740b-46e3-af41-65ad3f8d7057-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:36:32.743982 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.743962    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce82c784-9eb5-43cb-9f23-d971046fbaf7-plugins-config-volume" pod "ce82c784-9eb5-43cb-9f23-d971046fbaf7" (UID: "ce82c784-9eb5-43cb-9f23-d971046fbaf7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:36:32.745688 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.745653    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce82c784-9eb5-43cb-9f23-d971046fbaf7-kube-api-access-hcg96" pod "ce82c784-9eb5-43cb-9f23-d971046fbaf7" (UID: "ce82c784-9eb5-43cb-9f23-d971046fbaf7"). InnerVolumeSpecName "kube-api-access-hcg96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:32.793545 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.793516    4327 generic.go:356] "Generic (PLEG): container finished" podID="ce82c784-9eb5-43cb-9f23-d971046fbaf7" containerID="c823067778c221557e1dc79de9d881bc2d75e73e5887020198bd6e5605b19e4b" exitCode=0
Oct 31 01:36:32.793673 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.793563    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg"
Oct 31 01:36:32.793673 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.793602    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg" event={"ID":"ce82c784-9eb5-43cb-9f23-d971046fbaf7","Type":"ContainerDied","Data":"c823067778c221557e1dc79de9d881bc2d75e73e5887020198bd6e5605b19e4b"}
Oct 31 01:36:32.793673 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.793659    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg" event={"ID":"ce82c784-9eb5-43cb-9f23-d971046fbaf7","Type":"ContainerDied","Data":"4c675620e748c6420c2444656b0aab8fbb92acc73b182f1789dd510a7ca38368"}
Oct 31 01:36:32.793878 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.793702    4327 scope.go:122] "RemoveContainer" containerID="c823067778c221557e1dc79de9d881bc2d75e73e5887020198bd6e5605b19e4b"
Oct 31 01:36:32.794915 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.794896    4327 generic.go:356] "Generic (PLEG): container finished" podID="25a47e57-740b-46e3-af41-65ad3f8d7057" containerID="d0ad806c0f99eacbc1c36305392cba999330a7712a0557e252c530b142b71fc1" exitCode=0
Oct 31 01:36:32.795002 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.794934    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc" event={"ID":"25a47e57-740b-46e3-af41-65ad3f8d7057","Type":"ContainerDied","Data":"d0ad806c0f99eacbc1c36305392cba999330a7712a0557e252c530b142b71fc1"}
Oct 31 01:36:32.795002 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.794945    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc"
Oct 31 01:36:32.795002 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.794967    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc" event={"ID":"25a47e57-740b-46e3-af41-65ad3f8d7057","Type":"ContainerDied","Data":"f3eca892832a5a8dee8892ff9fa14f1ebd924bbcbd3ca78d9e12ea8cf1407b19"}
Oct 31 01:36:32.803054 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.803039    4327 scope.go:122] "RemoveContainer" containerID="c823067778c221557e1dc79de9d881bc2d75e73e5887020198bd6e5605b19e4b"
Oct 31 01:36:32.803328 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:36:32.803303    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c823067778c221557e1dc79de9d881bc2d75e73e5887020198bd6e5605b19e4b\": container with ID starting with c823067778c221557e1dc79de9d881bc2d75e73e5887020198bd6e5605b19e4b not found: ID does not exist" containerID="c823067778c221557e1dc79de9d881bc2d75e73e5887020198bd6e5605b19e4b"
Oct 31 01:36:32.803430 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.803339    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c823067778c221557e1dc79de9d881bc2d75e73e5887020198bd6e5605b19e4b"} err="failed to get container status \"c823067778c221557e1dc79de9d881bc2d75e73e5887020198bd6e5605b19e4b\": rpc error: code = NotFound desc = could not find container \"c823067778c221557e1dc79de9d881bc2d75e73e5887020198bd6e5605b19e4b\": container with ID starting with c823067778c221557e1dc79de9d881bc2d75e73e5887020198bd6e5605b19e4b not found: ID does not exist"
Oct 31 01:36:32.803430 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.803360    4327 scope.go:122] "RemoveContainer" containerID="d0ad806c0f99eacbc1c36305392cba999330a7712a0557e252c530b142b71fc1"
Oct 31 01:36:32.810911 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.810893    4327 scope.go:122] "RemoveContainer" containerID="d0ad806c0f99eacbc1c36305392cba999330a7712a0557e252c530b142b71fc1"
Oct 31 01:36:32.811181 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:36:32.811160    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0ad806c0f99eacbc1c36305392cba999330a7712a0557e252c530b142b71fc1\": container with ID starting with d0ad806c0f99eacbc1c36305392cba999330a7712a0557e252c530b142b71fc1 not found: ID does not exist" containerID="d0ad806c0f99eacbc1c36305392cba999330a7712a0557e252c530b142b71fc1"
Oct 31 01:36:32.811249 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.811187    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ad806c0f99eacbc1c36305392cba999330a7712a0557e252c530b142b71fc1"} err="failed to get container status \"d0ad806c0f99eacbc1c36305392cba999330a7712a0557e252c530b142b71fc1\": rpc error: code = NotFound desc = could not find container \"d0ad806c0f99eacbc1c36305392cba999330a7712a0557e252c530b142b71fc1\": container with ID starting with d0ad806c0f99eacbc1c36305392cba999330a7712a0557e252c530b142b71fc1 not found: ID does not exist"
Oct 31 01:36:32.844707 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.844627    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc"]
Oct 31 01:36:32.845096 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.845077    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcg96\" (UniqueName: \"kubernetes.io/projected/ce82c784-9eb5-43cb-9f23-d971046fbaf7-kube-api-access-hcg96\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:36:32.845160 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.845099    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce82c784-9eb5-43cb-9f23-d971046fbaf7-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:36:32.849270 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.849253    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2e17bf-cfc884b4528nwc"]
Oct 31 01:36:32.881066 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.881040    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg"]
Oct 31 01:36:32.885972 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:32.885949    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e17bf-b8894mhrbg"]
Oct 31 01:36:33.918513 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:33.918475    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25a47e57-740b-46e3-af41-65ad3f8d7057" path="/var/lib/kubelet/pods/25a47e57-740b-46e3-af41-65ad3f8d7057/volumes"
Oct 31 01:36:33.918922 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:33.918874    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce82c784-9eb5-43cb-9f23-d971046fbaf7" path="/var/lib/kubelet/pods/ce82c784-9eb5-43cb-9f23-d971046fbaf7/volumes"
Oct 31 01:36:37.659577 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.659544    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"]
Oct 31 01:36:37.660026 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.659972    4327 cpu_manager.go:414] "containerMap: removing container" podUID="8e929816-7500-4cc0-b2a4-eb50cb2f6be7" containerName="istio-proxy"
Oct 31 01:36:37.660026 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.659984    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e929816-7500-4cc0-b2a4-eb50cb2f6be7" containerName="istio-proxy"
Oct 31 01:36:37.660026 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.660009    4327 cpu_manager.go:414] "containerMap: removing container" podUID="ce82c784-9eb5-43cb-9f23-d971046fbaf7" containerName="payload-pre-processing"
Oct 31 01:36:37.660026 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.660014    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce82c784-9eb5-43cb-9f23-d971046fbaf7" containerName="payload-pre-processing"
Oct 31 01:36:37.660026 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.660021    4327 cpu_manager.go:414] "containerMap: removing container" podUID="25a47e57-740b-46e3-af41-65ad3f8d7057" containerName="payload-processing"
Oct 31 01:36:37.660026 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.660026    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25a47e57-740b-46e3-af41-65ad3f8d7057" containerName="payload-processing"
Oct 31 01:36:37.660263 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.660079    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e929816-7500-4cc0-b2a4-eb50cb2f6be7" containerName="istio-proxy"
Oct 31 01:36:37.660263 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.660089    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e929816-7500-4cc0-b2a4-eb50cb2f6be7" containerName="istio-proxy"
Oct 31 01:36:37.660263 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.660096    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce82c784-9eb5-43cb-9f23-d971046fbaf7" containerName="payload-pre-processing"
Oct 31 01:36:37.660263 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.660102    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="25a47e57-740b-46e3-af41-65ad3f8d7057" containerName="payload-processing"
Oct 31 01:36:37.664851 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.664828    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"
Oct 31 01:36:37.687302 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.687244    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-model-cache\") pod \"e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk\" (UID: \"4720045e-5b23-41af-8dbe-7bc50c0012a9\") " pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"
Oct 31 01:36:37.687574 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.687333    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-tmp-dir\") pod \"e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk\" (UID: \"4720045e-5b23-41af-8dbe-7bc50c0012a9\") " pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"
Oct 31 01:36:37.687574 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.687521    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-home\") pod \"e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk\" (UID: \"4720045e-5b23-41af-8dbe-7bc50c0012a9\") " pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"
Oct 31 01:36:37.687760 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.687637    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4720045e-5b23-41af-8dbe-7bc50c0012a9-tls-certs\") pod \"e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk\" (UID: \"4720045e-5b23-41af-8dbe-7bc50c0012a9\") " pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"
Oct 31 01:36:37.687760 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.687707    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-dshm\") pod \"e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk\" (UID: \"4720045e-5b23-41af-8dbe-7bc50c0012a9\") " pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"
Oct 31 01:36:37.687760 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.687748    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4mnm\" (UniqueName: \"kubernetes.io/projected/4720045e-5b23-41af-8dbe-7bc50c0012a9-kube-api-access-c4mnm\") pod \"e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk\" (UID: \"4720045e-5b23-41af-8dbe-7bc50c0012a9\") " pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"
Oct 31 01:36:37.691898 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.691876    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-51f40966-kserve-self-signed-certs\""
Oct 31 01:36:37.693703 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.693668    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"]
Oct 31 01:36:37.788858 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.788821    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-home\") pod \"e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk\" (UID: \"4720045e-5b23-41af-8dbe-7bc50c0012a9\") " pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"
Oct 31 01:36:37.788858 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.788864    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4720045e-5b23-41af-8dbe-7bc50c0012a9-tls-certs\") pod \"e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk\" (UID: \"4720045e-5b23-41af-8dbe-7bc50c0012a9\") " pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"
Oct 31 01:36:37.789120 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.788887    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-dshm\") pod \"e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk\" (UID: \"4720045e-5b23-41af-8dbe-7bc50c0012a9\") " pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"
Oct 31 01:36:37.789120 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.788919    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4mnm\" (UniqueName: \"kubernetes.io/projected/4720045e-5b23-41af-8dbe-7bc50c0012a9-kube-api-access-c4mnm\") pod \"e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk\" (UID: \"4720045e-5b23-41af-8dbe-7bc50c0012a9\") " pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"
Oct 31 01:36:37.789120 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.788947    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-model-cache\") pod \"e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk\" (UID: \"4720045e-5b23-41af-8dbe-7bc50c0012a9\") " pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"
Oct 31 01:36:37.789120 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.788973    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-tmp-dir\") pod \"e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk\" (UID: \"4720045e-5b23-41af-8dbe-7bc50c0012a9\") " pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"
Oct 31 01:36:37.789348 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.789221    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-home\") pod \"e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk\" (UID: \"4720045e-5b23-41af-8dbe-7bc50c0012a9\") " pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"
Oct 31 01:36:37.789348 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.789326    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-tmp-dir\") pod \"e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk\" (UID: \"4720045e-5b23-41af-8dbe-7bc50c0012a9\") " pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"
Oct 31 01:36:37.789430 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.789368    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-model-cache\") pod \"e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk\" (UID: \"4720045e-5b23-41af-8dbe-7bc50c0012a9\") " pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"
Oct 31 01:36:37.791210 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.791187    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-dshm\") pod \"e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk\" (UID: \"4720045e-5b23-41af-8dbe-7bc50c0012a9\") " pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"
Oct 31 01:36:37.791362 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.791322    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4720045e-5b23-41af-8dbe-7bc50c0012a9-tls-certs\") pod \"e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk\" (UID: \"4720045e-5b23-41af-8dbe-7bc50c0012a9\") " pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"
Oct 31 01:36:37.812234 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.812202    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4mnm\" (UniqueName: \"kubernetes.io/projected/4720045e-5b23-41af-8dbe-7bc50c0012a9-kube-api-access-c4mnm\") pod \"e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk\" (UID: \"4720045e-5b23-41af-8dbe-7bc50c0012a9\") " pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"
Oct 31 01:36:37.978323 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:37.978240    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"
Oct 31 01:36:38.284931 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:38.284903    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"]
Oct 31 01:36:38.287653 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:36:38.287626    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4720045e_5b23_41af_8dbe_7bc50c0012a9.slice/crio-5babf430196ba1cb598bbc4a3dfda4248656fa41ae9d2034e5c9caa6c935ff5c WatchSource:0}: Error finding container 5babf430196ba1cb598bbc4a3dfda4248656fa41ae9d2034e5c9caa6c935ff5c: Status 404 returned error can't find the container with id 5babf430196ba1cb598bbc4a3dfda4248656fa41ae9d2034e5c9caa6c935ff5c
Oct 31 01:36:38.825700 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:38.825631    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk" event={"ID":"4720045e-5b23-41af-8dbe-7bc50c0012a9","Type":"ContainerStarted","Data":"91c2fa864ea55e98a0b0faa05847b0b0cedc9ca9035e891377be689444207904"}
Oct 31 01:36:38.826118 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:38.825717    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk" event={"ID":"4720045e-5b23-41af-8dbe-7bc50c0012a9","Type":"ContainerStarted","Data":"5babf430196ba1cb598bbc4a3dfda4248656fa41ae9d2034e5c9caa6c935ff5c"}
Oct 31 01:36:38.870530 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:38.870479    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk" podStartSLOduration=1.870465583 podStartE2EDuration="1.870465583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:36:38.868812442 +0000 UTC m=+4659.090847997" watchObservedRunningTime="2026-10-31 01:36:38.870465583 +0000 UTC m=+4659.092501124"
Oct 31 01:36:38.978711 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:38.978656    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"
Oct 31 01:36:45.291781 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:36:45.291737    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:36:47.978561 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:47.978526    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"
Oct 31 01:36:47.991159 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:47.991132    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"
Oct 31 01:36:48.903569 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:36:48.903537    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"
Oct 31 01:37:15.294910 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:37:15.294863    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:37:16.055391 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:16.054137    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"]
Oct 31 01:37:31.218518 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:31.218475    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk" podUID="4720045e-5b23-41af-8dbe-7bc50c0012a9" containerName="main" containerID="cri-o://91c2fa864ea55e98a0b0faa05847b0b0cedc9ca9035e891377be689444207904" gracePeriod=45
Oct 31 01:37:31.557202 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:31.556866    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"
Oct 31 01:37:31.605892 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:31.605859    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-home\" (UniqueName: \"kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-home\") pod \"4720045e-5b23-41af-8dbe-7bc50c0012a9\" (UID: \"4720045e-5b23-41af-8dbe-7bc50c0012a9\") "
Oct 31 01:37:31.606062 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:31.605936    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-model-cache\") pod \"4720045e-5b23-41af-8dbe-7bc50c0012a9\" (UID: \"4720045e-5b23-41af-8dbe-7bc50c0012a9\") "
Oct 31 01:37:31.606062 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:31.605964    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4720045e-5b23-41af-8dbe-7bc50c0012a9-kube-api-access-c4mnm\" (UniqueName: \"kubernetes.io/projected/4720045e-5b23-41af-8dbe-7bc50c0012a9-kube-api-access-c4mnm\") pod \"4720045e-5b23-41af-8dbe-7bc50c0012a9\" (UID: \"4720045e-5b23-41af-8dbe-7bc50c0012a9\") "
Oct 31 01:37:31.606062 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:31.605989    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-tmp-dir\") pod \"4720045e-5b23-41af-8dbe-7bc50c0012a9\" (UID: \"4720045e-5b23-41af-8dbe-7bc50c0012a9\") "
Oct 31 01:37:31.606062 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:31.606006    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-dshm\") pod \"4720045e-5b23-41af-8dbe-7bc50c0012a9\" (UID: \"4720045e-5b23-41af-8dbe-7bc50c0012a9\") "
Oct 31 01:37:31.606302 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:31.606111    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4720045e-5b23-41af-8dbe-7bc50c0012a9-tls-certs\" (UniqueName: \"kubernetes.io/secret/4720045e-5b23-41af-8dbe-7bc50c0012a9-tls-certs\") pod \"4720045e-5b23-41af-8dbe-7bc50c0012a9\" (UID: \"4720045e-5b23-41af-8dbe-7bc50c0012a9\") "
Oct 31 01:37:31.606302 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:31.606154    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-home" pod "4720045e-5b23-41af-8dbe-7bc50c0012a9" (UID: "4720045e-5b23-41af-8dbe-7bc50c0012a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:37:31.606302 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:31.606237    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-tmp-dir" pod "4720045e-5b23-41af-8dbe-7bc50c0012a9" (UID: "4720045e-5b23-41af-8dbe-7bc50c0012a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:37:31.606302 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:31.606249    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-model-cache" pod "4720045e-5b23-41af-8dbe-7bc50c0012a9" (UID: "4720045e-5b23-41af-8dbe-7bc50c0012a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:37:31.606515 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:31.606445    4327 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-home\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:37:31.606515 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:31.606463    4327 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-model-cache\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:37:31.606515 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:31.606478    4327 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-tmp-dir\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:37:31.608266 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:31.608236    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4720045e-5b23-41af-8dbe-7bc50c0012a9-tls-certs" pod "4720045e-5b23-41af-8dbe-7bc50c0012a9" (UID: "4720045e-5b23-41af-8dbe-7bc50c0012a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:37:31.608266 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:31.608240    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-dshm" pod "4720045e-5b23-41af-8dbe-7bc50c0012a9" (UID: "4720045e-5b23-41af-8dbe-7bc50c0012a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:37:31.610936 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:31.610912    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4720045e-5b23-41af-8dbe-7bc50c0012a9-kube-api-access-c4mnm" pod "4720045e-5b23-41af-8dbe-7bc50c0012a9" (UID: "4720045e-5b23-41af-8dbe-7bc50c0012a9"). InnerVolumeSpecName "kube-api-access-c4mnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:37:31.707325 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:31.707290    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4mnm\" (UniqueName: \"kubernetes.io/projected/4720045e-5b23-41af-8dbe-7bc50c0012a9-kube-api-access-c4mnm\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:37:31.707325 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:31.707322    4327 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4720045e-5b23-41af-8dbe-7bc50c0012a9-dshm\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:37:31.707534 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:31.707335    4327 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4720045e-5b23-41af-8dbe-7bc50c0012a9-tls-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:37:32.265283 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:32.263158    4327 generic.go:356] "Generic (PLEG): container finished" podID="4720045e-5b23-41af-8dbe-7bc50c0012a9" containerID="91c2fa864ea55e98a0b0faa05847b0b0cedc9ca9035e891377be689444207904" exitCode=0
Oct 31 01:37:32.265283 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:32.263298    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk" event={"ID":"4720045e-5b23-41af-8dbe-7bc50c0012a9","Type":"ContainerDied","Data":"91c2fa864ea55e98a0b0faa05847b0b0cedc9ca9035e891377be689444207904"}
Oct 31 01:37:32.265283 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:32.263340    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk" event={"ID":"4720045e-5b23-41af-8dbe-7bc50c0012a9","Type":"ContainerDied","Data":"5babf430196ba1cb598bbc4a3dfda4248656fa41ae9d2034e5c9caa6c935ff5c"}
Oct 31 01:37:32.265283 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:32.263365    4327 scope.go:122] "RemoveContainer" containerID="91c2fa864ea55e98a0b0faa05847b0b0cedc9ca9035e891377be689444207904"
Oct 31 01:37:32.265283 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:32.263511    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"
Oct 31 01:37:32.302423 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:32.302388    4327 scope.go:122] "RemoveContainer" containerID="91c2fa864ea55e98a0b0faa05847b0b0cedc9ca9035e891377be689444207904"
Oct 31 01:37:32.302935 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:37:32.302905    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91c2fa864ea55e98a0b0faa05847b0b0cedc9ca9035e891377be689444207904\": container with ID starting with 91c2fa864ea55e98a0b0faa05847b0b0cedc9ca9035e891377be689444207904 not found: ID does not exist" containerID="91c2fa864ea55e98a0b0faa05847b0b0cedc9ca9035e891377be689444207904"
Oct 31 01:37:32.303029 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:32.302948    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c2fa864ea55e98a0b0faa05847b0b0cedc9ca9035e891377be689444207904"} err="failed to get container status \"91c2fa864ea55e98a0b0faa05847b0b0cedc9ca9035e891377be689444207904\": rpc error: code = NotFound desc = could not find container \"91c2fa864ea55e98a0b0faa05847b0b0cedc9ca9035e891377be689444207904\": container with ID starting with 91c2fa864ea55e98a0b0faa05847b0b0cedc9ca9035e891377be689444207904 not found: ID does not exist"
Oct 31 01:37:32.339711 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:32.338584    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"]
Oct 31 01:37:32.367705 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:32.353081    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-51f40966-kserve-6cf49bb54f-p5prk"]
Oct 31 01:37:33.919946 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:33.919910    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4720045e-5b23-41af-8dbe-7bc50c0012a9" path="/var/lib/kubelet/pods/4720045e-5b23-41af-8dbe-7bc50c0012a9/volumes"
Oct 31 01:37:45.292308 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:37:45.292266    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:37:55.823472 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:55.823437    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-749d798f87-t7cvd"]
Oct 31 01:37:55.824026 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:55.823849    4327 cpu_manager.go:414] "containerMap: removing container" podUID="4720045e-5b23-41af-8dbe-7bc50c0012a9" containerName="main"
Oct 31 01:37:55.824026 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:55.823863    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4720045e-5b23-41af-8dbe-7bc50c0012a9" containerName="main"
Oct 31 01:37:55.824026 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:55.823886    4327 cpu_manager.go:414] "containerMap: removing container" podUID="8e929816-7500-4cc0-b2a4-eb50cb2f6be7" containerName="istio-proxy"
Oct 31 01:37:55.824026 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:55.823891    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e929816-7500-4cc0-b2a4-eb50cb2f6be7" containerName="istio-proxy"
Oct 31 01:37:55.824026 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:55.823949    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="4720045e-5b23-41af-8dbe-7bc50c0012a9" containerName="main"
Oct 31 01:37:55.827233 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:55.827205    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-749d798f87-t7cvd"
Oct 31 01:37:55.857053 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:55.857025    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-749d798f87-t7cvd"]
Oct 31 01:37:55.992883 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:55.992848    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/900c9d95-ccc6-4663-b994-2c4a2d3188c6-maas-api-tls\") pod \"maas-api-749d798f87-t7cvd\" (UID: \"900c9d95-ccc6-4663-b994-2c4a2d3188c6\") " pod="odh-ai-gateway-infra/maas-api-749d798f87-t7cvd"
Oct 31 01:37:55.993066 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:55.992928    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d2dk\" (UniqueName: \"kubernetes.io/projected/900c9d95-ccc6-4663-b994-2c4a2d3188c6-kube-api-access-9d2dk\") pod \"maas-api-749d798f87-t7cvd\" (UID: \"900c9d95-ccc6-4663-b994-2c4a2d3188c6\") " pod="odh-ai-gateway-infra/maas-api-749d798f87-t7cvd"
Oct 31 01:37:55.993066 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:55.992967    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/900c9d95-ccc6-4663-b994-2c4a2d3188c6-metrics-certs\") pod \"maas-api-749d798f87-t7cvd\" (UID: \"900c9d95-ccc6-4663-b994-2c4a2d3188c6\") " pod="odh-ai-gateway-infra/maas-api-749d798f87-t7cvd"
Oct 31 01:37:55.993164 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:55.993088    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/900c9d95-ccc6-4663-b994-2c4a2d3188c6-openshift-service-ca\") pod \"maas-api-749d798f87-t7cvd\" (UID: \"900c9d95-ccc6-4663-b994-2c4a2d3188c6\") " pod="odh-ai-gateway-infra/maas-api-749d798f87-t7cvd"
Oct 31 01:37:56.093741 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:56.093650    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/900c9d95-ccc6-4663-b994-2c4a2d3188c6-maas-api-tls\") pod \"maas-api-749d798f87-t7cvd\" (UID: \"900c9d95-ccc6-4663-b994-2c4a2d3188c6\") " pod="odh-ai-gateway-infra/maas-api-749d798f87-t7cvd"
Oct 31 01:37:56.093916 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:56.093789    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d2dk\" (UniqueName: \"kubernetes.io/projected/900c9d95-ccc6-4663-b994-2c4a2d3188c6-kube-api-access-9d2dk\") pod \"maas-api-749d798f87-t7cvd\" (UID: \"900c9d95-ccc6-4663-b994-2c4a2d3188c6\") " pod="odh-ai-gateway-infra/maas-api-749d798f87-t7cvd"
Oct 31 01:37:56.093916 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:56.093837    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/900c9d95-ccc6-4663-b994-2c4a2d3188c6-metrics-certs\") pod \"maas-api-749d798f87-t7cvd\" (UID: \"900c9d95-ccc6-4663-b994-2c4a2d3188c6\") " pod="odh-ai-gateway-infra/maas-api-749d798f87-t7cvd"
Oct 31 01:37:56.093916 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:56.093913    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/900c9d95-ccc6-4663-b994-2c4a2d3188c6-openshift-service-ca\") pod \"maas-api-749d798f87-t7cvd\" (UID: \"900c9d95-ccc6-4663-b994-2c4a2d3188c6\") " pod="odh-ai-gateway-infra/maas-api-749d798f87-t7cvd"
Oct 31 01:37:56.094491 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:56.094469    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/900c9d95-ccc6-4663-b994-2c4a2d3188c6-openshift-service-ca\") pod \"maas-api-749d798f87-t7cvd\" (UID: \"900c9d95-ccc6-4663-b994-2c4a2d3188c6\") " pod="odh-ai-gateway-infra/maas-api-749d798f87-t7cvd"
Oct 31 01:37:56.096069 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:56.096047    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/900c9d95-ccc6-4663-b994-2c4a2d3188c6-maas-api-tls\") pod \"maas-api-749d798f87-t7cvd\" (UID: \"900c9d95-ccc6-4663-b994-2c4a2d3188c6\") " pod="odh-ai-gateway-infra/maas-api-749d798f87-t7cvd"
Oct 31 01:37:56.096189 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:56.096171    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/900c9d95-ccc6-4663-b994-2c4a2d3188c6-metrics-certs\") pod \"maas-api-749d798f87-t7cvd\" (UID: \"900c9d95-ccc6-4663-b994-2c4a2d3188c6\") " pod="odh-ai-gateway-infra/maas-api-749d798f87-t7cvd"
Oct 31 01:37:56.116824 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:56.116799    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d2dk\" (UniqueName: \"kubernetes.io/projected/900c9d95-ccc6-4663-b994-2c4a2d3188c6-kube-api-access-9d2dk\") pod \"maas-api-749d798f87-t7cvd\" (UID: \"900c9d95-ccc6-4663-b994-2c4a2d3188c6\") " pod="odh-ai-gateway-infra/maas-api-749d798f87-t7cvd"
Oct 31 01:37:56.143895 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:56.143870    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-749d798f87-t7cvd"
Oct 31 01:37:56.439201 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:56.439179    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-749d798f87-t7cvd"]
Oct 31 01:37:56.441591 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:37:56.441562    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod900c9d95_ccc6_4663_b994_2c4a2d3188c6.slice/crio-b2f9152ce3bedc95c4ae38f3c3dec58a6bbf29431abd5960c4a887939464b1aa WatchSource:0}: Error finding container b2f9152ce3bedc95c4ae38f3c3dec58a6bbf29431abd5960c4a887939464b1aa: Status 404 returned error can't find the container with id b2f9152ce3bedc95c4ae38f3c3dec58a6bbf29431abd5960c4a887939464b1aa
Oct 31 01:37:56.442840 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:56.442821    4327 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:37:57.397548 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:57.397509    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-749d798f87-t7cvd" event={"ID":"900c9d95-ccc6-4663-b994-2c4a2d3188c6","Type":"ContainerStarted","Data":"335eacde2dc9f7ec73a67be4bd4b6ea7243d4e3b3981939c4c46a7580ed271c5"}
Oct 31 01:37:57.397962 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:57.397591    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-749d798f87-t7cvd" event={"ID":"900c9d95-ccc6-4663-b994-2c4a2d3188c6","Type":"ContainerStarted","Data":"b2f9152ce3bedc95c4ae38f3c3dec58a6bbf29431abd5960c4a887939464b1aa"}
Oct 31 01:37:57.397962 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:57.397620    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-749d798f87-t7cvd"
Oct 31 01:37:57.440359 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:37:57.440310    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-749d798f87-t7cvd" podStartSLOduration=1.985011069 podStartE2EDuration="2.440297157s" totalImagesPullingTime="455.286088ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:37:57.439634925 +0000 UTC m=+4737.661670467" watchObservedRunningTime="2026-10-31 01:37:57.440297157 +0000 UTC m=+4737.662332700"
Oct 31 01:38:03.407921 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:03.407892    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-749d798f87-t7cvd"
Oct 31 01:38:03.496469 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:03.496428    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8"]
Oct 31 01:38:03.496746 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:03.496712    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8" podUID="de8f47da-a2b5-4b42-825c-fd08144e9028" containerName="maas-api" containerID="cri-o://232b23d8c027effd696dcd5800809cff4515ea3f16c2a0023866fd0ea39eceec" gracePeriod=30
Oct 31 01:38:03.708356 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:03.708273    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8" podUID="de8f47da-a2b5-4b42-825c-fd08144e9028" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.28:8443/health\": dial tcp 10.133.0.28:8443: connect: connection refused"
Oct 31 01:38:04.745040 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:04.745016    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8"
Oct 31 01:38:04.818269 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:04.818190    4327 scope.go:122] "RemoveContainer" containerID="232b23d8c027effd696dcd5800809cff4515ea3f16c2a0023866fd0ea39eceec"
Oct 31 01:38:04.879442 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:04.879417    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de8f47da-a2b5-4b42-825c-fd08144e9028-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de8f47da-a2b5-4b42-825c-fd08144e9028-metrics-certs\") pod \"de8f47da-a2b5-4b42-825c-fd08144e9028\" (UID: \"de8f47da-a2b5-4b42-825c-fd08144e9028\") "
Oct 31 01:38:04.879567 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:04.879452    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de8f47da-a2b5-4b42-825c-fd08144e9028-kube-api-access-ssjml\" (UniqueName: \"kubernetes.io/projected/de8f47da-a2b5-4b42-825c-fd08144e9028-kube-api-access-ssjml\") pod \"de8f47da-a2b5-4b42-825c-fd08144e9028\" (UID: \"de8f47da-a2b5-4b42-825c-fd08144e9028\") "
Oct 31 01:38:04.879567 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:04.879482    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de8f47da-a2b5-4b42-825c-fd08144e9028-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de8f47da-a2b5-4b42-825c-fd08144e9028-openshift-service-ca\") pod \"de8f47da-a2b5-4b42-825c-fd08144e9028\" (UID: \"de8f47da-a2b5-4b42-825c-fd08144e9028\") "
Oct 31 01:38:04.879567 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:04.879549    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de8f47da-a2b5-4b42-825c-fd08144e9028-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de8f47da-a2b5-4b42-825c-fd08144e9028-maas-api-tls\") pod \"de8f47da-a2b5-4b42-825c-fd08144e9028\" (UID: \"de8f47da-a2b5-4b42-825c-fd08144e9028\") "
Oct 31 01:38:04.879960 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:04.879932    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de8f47da-a2b5-4b42-825c-fd08144e9028-openshift-service-ca" pod "de8f47da-a2b5-4b42-825c-fd08144e9028" (UID: "de8f47da-a2b5-4b42-825c-fd08144e9028"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:38:04.881553 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:04.881528    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de8f47da-a2b5-4b42-825c-fd08144e9028-metrics-certs" pod "de8f47da-a2b5-4b42-825c-fd08144e9028" (UID: "de8f47da-a2b5-4b42-825c-fd08144e9028"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:38:04.881668 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:04.881599    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de8f47da-a2b5-4b42-825c-fd08144e9028-kube-api-access-ssjml" pod "de8f47da-a2b5-4b42-825c-fd08144e9028" (UID: "de8f47da-a2b5-4b42-825c-fd08144e9028"). InnerVolumeSpecName "kube-api-access-ssjml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:38:04.881767 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:04.881749    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de8f47da-a2b5-4b42-825c-fd08144e9028-maas-api-tls" pod "de8f47da-a2b5-4b42-825c-fd08144e9028" (UID: "de8f47da-a2b5-4b42-825c-fd08144e9028"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:38:04.980918 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:04.980893    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de8f47da-a2b5-4b42-825c-fd08144e9028-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:38:04.980918 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:04.980916    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de8f47da-a2b5-4b42-825c-fd08144e9028-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:38:04.981104 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:04.980926    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssjml\" (UniqueName: \"kubernetes.io/projected/de8f47da-a2b5-4b42-825c-fd08144e9028-kube-api-access-ssjml\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:38:04.981104 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:04.980937    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de8f47da-a2b5-4b42-825c-fd08144e9028-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:38:05.433570 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:05.433538    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8"
Oct 31 01:38:05.433773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:05.433542    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8" event={"ID":"de8f47da-a2b5-4b42-825c-fd08144e9028","Type":"ContainerDied","Data":"232b23d8c027effd696dcd5800809cff4515ea3f16c2a0023866fd0ea39eceec"}
Oct 31 01:38:05.433773 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:05.433668    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8" event={"ID":"de8f47da-a2b5-4b42-825c-fd08144e9028","Type":"ContainerDied","Data":"af0e134fa6a67c2046c64b14cacfb33f692ef95d5f09621d546aab9dbc25b39a"}
Oct 31 01:38:05.512293 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:05.512260    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8"]
Oct 31 01:38:05.516711 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:05.516666    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fb4ff9887-splg8"]
Oct 31 01:38:05.919778 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:05.919746    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de8f47da-a2b5-4b42-825c-fd08144e9028" path="/var/lib/kubelet/pods/de8f47da-a2b5-4b42-825c-fd08144e9028/volumes"
Oct 31 01:38:09.132554 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.132519    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv"]
Oct 31 01:38:09.133056 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.132899    4327 cpu_manager.go:414] "containerMap: removing container" podUID="de8f47da-a2b5-4b42-825c-fd08144e9028" containerName="maas-api"
Oct 31 01:38:09.133056 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.132911    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de8f47da-a2b5-4b42-825c-fd08144e9028" containerName="maas-api"
Oct 31 01:38:09.133056 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.132991    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="de8f47da-a2b5-4b42-825c-fd08144e9028" containerName="maas-api"
Oct 31 01:38:09.143873 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.143846    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv"
Oct 31 01:38:09.167699 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.161391    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv"]
Oct 31 01:38:09.167893 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.167848    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1c206c91-serving-cert\""
Oct 31 01:38:09.224118 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.224077    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-openshift-service-ca\") pod \"maas-api-e2e-ait-1c206c91-7d48865874-plzwv\" (UID: \"eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv"
Oct 31 01:38:09.224317 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.224134    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5knpz\" (UniqueName: \"kubernetes.io/projected/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-kube-api-access-5knpz\") pod \"maas-api-e2e-ait-1c206c91-7d48865874-plzwv\" (UID: \"eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv"
Oct 31 01:38:09.224317 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.224233    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-metrics-certs\") pod \"maas-api-e2e-ait-1c206c91-7d48865874-plzwv\" (UID: \"eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv"
Oct 31 01:38:09.224434 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.224367    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-maas-api-tls\") pod \"maas-api-e2e-ait-1c206c91-7d48865874-plzwv\" (UID: \"eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv"
Oct 31 01:38:09.325699 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.325645    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-maas-api-tls\") pod \"maas-api-e2e-ait-1c206c91-7d48865874-plzwv\" (UID: \"eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv"
Oct 31 01:38:09.325890 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.325731    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-openshift-service-ca\") pod \"maas-api-e2e-ait-1c206c91-7d48865874-plzwv\" (UID: \"eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv"
Oct 31 01:38:09.325890 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.325775    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5knpz\" (UniqueName: \"kubernetes.io/projected/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-kube-api-access-5knpz\") pod \"maas-api-e2e-ait-1c206c91-7d48865874-plzwv\" (UID: \"eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv"
Oct 31 01:38:09.325890 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:38:09.325792    4327 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-serving-cert: secret "maas-api-e2e-ait-1c206c91-serving-cert" not found
Oct 31 01:38:09.325890 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.325812    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-metrics-certs\") pod \"maas-api-e2e-ait-1c206c91-7d48865874-plzwv\" (UID: \"eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv"
Oct 31 01:38:09.325890 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:38:09.325861    4327 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-maas-api-tls podName:eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e nodeName:}" failed. No retries permitted until 2026-10-31 01:38:09.825844807 +0000 UTC m=+4750.047880339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-maas-api-tls") pod "maas-api-e2e-ait-1c206c91-7d48865874-plzwv" (UID: "eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e") : secret "maas-api-e2e-ait-1c206c91-serving-cert" not found
Oct 31 01:38:09.326372 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.326353    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-openshift-service-ca\") pod \"maas-api-e2e-ait-1c206c91-7d48865874-plzwv\" (UID: \"eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv"
Oct 31 01:38:09.328318 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.328292    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-metrics-certs\") pod \"maas-api-e2e-ait-1c206c91-7d48865874-plzwv\" (UID: \"eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv"
Oct 31 01:38:09.349846 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.349816    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5knpz\" (UniqueName: \"kubernetes.io/projected/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-kube-api-access-5knpz\") pod \"maas-api-e2e-ait-1c206c91-7d48865874-plzwv\" (UID: \"eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv"
Oct 31 01:38:09.365815 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.365786    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8"]
Oct 31 01:38:09.369473 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.369458    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8"
Oct 31 01:38:09.384414 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.384360    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1c206c91-dockercfg-jgcqx\""
Oct 31 01:38:09.384591 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.384572    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1c206c91\""
Oct 31 01:38:09.392329 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.392304    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8"]
Oct 31 01:38:09.426537 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.426512    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0034165d-9b86-4556-806e-f64fc9156adc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8\" (UID: \"0034165d-9b86-4556-806e-f64fc9156adc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8"
Oct 31 01:38:09.426718 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.426578    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc6c8\" (UniqueName: \"kubernetes.io/projected/0034165d-9b86-4556-806e-f64fc9156adc-kube-api-access-wc6c8\") pod \"payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8\" (UID: \"0034165d-9b86-4556-806e-f64fc9156adc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8"
Oct 31 01:38:09.446889 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.446864    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp"]
Oct 31 01:38:09.450466 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.450443    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp"
Oct 31 01:38:09.470210 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.470182    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp"]
Oct 31 01:38:09.527434 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.527386    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dvqc\" (UniqueName: \"kubernetes.io/projected/cc03e2b4-0a09-43ff-8f69-0ba0a590123f-kube-api-access-9dvqc\") pod \"payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp\" (UID: \"cc03e2b4-0a09-43ff-8f69-0ba0a590123f\") " pod="openshift-ingress/payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp"
Oct 31 01:38:09.527654 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.527466    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0034165d-9b86-4556-806e-f64fc9156adc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8\" (UID: \"0034165d-9b86-4556-806e-f64fc9156adc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8"
Oct 31 01:38:09.527654 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.527530    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc03e2b4-0a09-43ff-8f69-0ba0a590123f-plugins-config-volume\") pod \"payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp\" (UID: \"cc03e2b4-0a09-43ff-8f69-0ba0a590123f\") " pod="openshift-ingress/payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp"
Oct 31 01:38:09.527654 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.527564    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc6c8\" (UniqueName: \"kubernetes.io/projected/0034165d-9b86-4556-806e-f64fc9156adc-kube-api-access-wc6c8\") pod \"payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8\" (UID: \"0034165d-9b86-4556-806e-f64fc9156adc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8"
Oct 31 01:38:09.529248 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.529217    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0034165d-9b86-4556-806e-f64fc9156adc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8\" (UID: \"0034165d-9b86-4556-806e-f64fc9156adc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8"
Oct 31 01:38:09.554019 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.553984    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc6c8\" (UniqueName: \"kubernetes.io/projected/0034165d-9b86-4556-806e-f64fc9156adc-kube-api-access-wc6c8\") pod \"payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8\" (UID: \"0034165d-9b86-4556-806e-f64fc9156adc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8"
Oct 31 01:38:09.628668 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.628629    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc03e2b4-0a09-43ff-8f69-0ba0a590123f-plugins-config-volume\") pod \"payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp\" (UID: \"cc03e2b4-0a09-43ff-8f69-0ba0a590123f\") " pod="openshift-ingress/payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp"
Oct 31 01:38:09.628877 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.628735    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dvqc\" (UniqueName: \"kubernetes.io/projected/cc03e2b4-0a09-43ff-8f69-0ba0a590123f-kube-api-access-9dvqc\") pod \"payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp\" (UID: \"cc03e2b4-0a09-43ff-8f69-0ba0a590123f\") " pod="openshift-ingress/payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp"
Oct 31 01:38:09.629321 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.629300    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc03e2b4-0a09-43ff-8f69-0ba0a590123f-plugins-config-volume\") pod \"payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp\" (UID: \"cc03e2b4-0a09-43ff-8f69-0ba0a590123f\") " pod="openshift-ingress/payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp"
Oct 31 01:38:09.652423 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.652358    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dvqc\" (UniqueName: \"kubernetes.io/projected/cc03e2b4-0a09-43ff-8f69-0ba0a590123f-kube-api-access-9dvqc\") pod \"payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp\" (UID: \"cc03e2b4-0a09-43ff-8f69-0ba0a590123f\") " pod="openshift-ingress/payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp"
Oct 31 01:38:09.683080 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.683057    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8"
Oct 31 01:38:09.763235 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.763207    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp"
Oct 31 01:38:09.831479 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.831453    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-maas-api-tls\") pod \"maas-api-e2e-ait-1c206c91-7d48865874-plzwv\" (UID: \"eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv"
Oct 31 01:38:09.833667 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.833641    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-maas-api-tls\") pod \"maas-api-e2e-ait-1c206c91-7d48865874-plzwv\" (UID: \"eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv"
Oct 31 01:38:09.979151 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:09.979122    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8"]
Oct 31 01:38:09.981192 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:38:09.981163    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0034165d_9b86_4556_806e_f64fc9156adc.slice/crio-aa36212aaaf46ba6f79869fa61a3dd1e606504a2d4ef96b7033db3caefc33746 WatchSource:0}: Error finding container aa36212aaaf46ba6f79869fa61a3dd1e606504a2d4ef96b7033db3caefc33746: Status 404 returned error can't find the container with id aa36212aaaf46ba6f79869fa61a3dd1e606504a2d4ef96b7033db3caefc33746
Oct 31 01:38:10.057177 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:10.057153    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv"
Oct 31 01:38:10.063863 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:10.063842    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp"]
Oct 31 01:38:10.066806 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:38:10.066780    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc03e2b4_0a09_43ff_8f69_0ba0a590123f.slice/crio-972b4a7749d365d0067ce755e8113bd2f31b94bcf824dddbd32b1e2916c62636 WatchSource:0}: Error finding container 972b4a7749d365d0067ce755e8113bd2f31b94bcf824dddbd32b1e2916c62636: Status 404 returned error can't find the container with id 972b4a7749d365d0067ce755e8113bd2f31b94bcf824dddbd32b1e2916c62636
Oct 31 01:38:10.370948 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:10.370923    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv"]
Oct 31 01:38:10.373278 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:38:10.373254    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb01ca4a_be4f_4fc3_ae20_870e5d0aff0e.slice/crio-41739f3f8bd99c151228bc5568c59a9dbbb73cb2837e7bd9fd28b87bc5df8544 WatchSource:0}: Error finding container 41739f3f8bd99c151228bc5568c59a9dbbb73cb2837e7bd9fd28b87bc5df8544: Status 404 returned error can't find the container with id 41739f3f8bd99c151228bc5568c59a9dbbb73cb2837e7bd9fd28b87bc5df8544
Oct 31 01:38:10.458525 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:10.458498    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8" event={"ID":"0034165d-9b86-4556-806e-f64fc9156adc","Type":"ContainerStarted","Data":"d40984d4fa23459fe67fa5b0b796f44f5096f4d998d3c19daeba021c5659f97b"}
Oct 31 01:38:10.458710 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:10.458555    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8" event={"ID":"0034165d-9b86-4556-806e-f64fc9156adc","Type":"ContainerStarted","Data":"aa36212aaaf46ba6f79869fa61a3dd1e606504a2d4ef96b7033db3caefc33746"}
Oct 31 01:38:10.458710 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:10.458693    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8"
Oct 31 01:38:10.460143 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:10.460121    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp" event={"ID":"cc03e2b4-0a09-43ff-8f69-0ba0a590123f","Type":"ContainerStarted","Data":"944cc1b722f9f622a13b72093d649c76a3561feac2b02b9dfb22d78ceeddcc1f"}
Oct 31 01:38:10.460260 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:10.460160    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp" event={"ID":"cc03e2b4-0a09-43ff-8f69-0ba0a590123f","Type":"ContainerStarted","Data":"972b4a7749d365d0067ce755e8113bd2f31b94bcf824dddbd32b1e2916c62636"}
Oct 31 01:38:10.460260 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:10.460188    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp"
Oct 31 01:38:10.461323 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:10.461305    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv" event={"ID":"eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e","Type":"ContainerStarted","Data":"41739f3f8bd99c151228bc5568c59a9dbbb73cb2837e7bd9fd28b87bc5df8544"}
Oct 31 01:38:10.499194 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:10.499140    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8" podStartSLOduration=1.499126201 podStartE2EDuration="1.499126201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:10.498203723 +0000 UTC m=+4750.720239266" watchObservedRunningTime="2026-10-31 01:38:10.499126201 +0000 UTC m=+4750.721161743"
Oct 31 01:38:10.540872 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:10.540809    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp" podStartSLOduration=1.540791647 podStartE2EDuration="1.540791647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:10.53973624 +0000 UTC m=+4750.761771783" watchObservedRunningTime="2026-10-31 01:38:10.540791647 +0000 UTC m=+4750.762827184"
Oct 31 01:38:11.469473 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:11.469434    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv" event={"ID":"eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e","Type":"ContainerStarted","Data":"e1f38c9eacef5ac335ba68dc290d768fb6193999eea1f678ba34a59453fbda44"}
Oct 31 01:38:11.469928 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:11.469716    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv"
Oct 31 01:38:11.512440 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:11.512392    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv" podStartSLOduration=2.203608015 podStartE2EDuration="2.512378708s" totalImagesPullingTime="308.770693ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:11.511623488 +0000 UTC m=+4751.733659030" watchObservedRunningTime="2026-10-31 01:38:11.512378708 +0000 UTC m=+4751.734414289"
Oct 31 01:38:15.291625 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:38:15.291583    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:38:17.479374 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:17.479340    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv"
Oct 31 01:38:21.472584 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:21.472556    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8"
Oct 31 01:38:21.472999 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:21.472623    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp"
Oct 31 01:38:32.689667 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:32.689626    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp"]
Oct 31 01:38:32.727823 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:32.727789    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8"]
Oct 31 01:38:34.717234 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:34.717198    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv"]
Oct 31 01:38:34.717766 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:34.717439    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv" podUID="eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e" containerName="maas-api" containerID="cri-o://e1f38c9eacef5ac335ba68dc290d768fb6193999eea1f678ba34a59453fbda44" gracePeriod=30
Oct 31 01:38:34.985837 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:34.985813    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv"
Oct 31 01:38:35.077274 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:35.077243    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-metrics-certs\") pod \"eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e\" (UID: \"eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e\") "
Oct 31 01:38:35.077450 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:35.077299    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-openshift-service-ca\") pod \"eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e\" (UID: \"eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e\") "
Oct 31 01:38:35.077450 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:35.077336    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-kube-api-access-5knpz\" (UniqueName: \"kubernetes.io/projected/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-kube-api-access-5knpz\") pod \"eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e\" (UID: \"eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e\") "
Oct 31 01:38:35.077450 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:35.077378    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-maas-api-tls\") pod \"eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e\" (UID: \"eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e\") "
Oct 31 01:38:35.077783 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:35.077754    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-openshift-service-ca" pod "eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e" (UID: "eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:38:35.079630 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:35.079598    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-maas-api-tls" pod "eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e" (UID: "eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:38:35.079763 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:35.079714    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-metrics-certs" pod "eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e" (UID: "eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:38:35.079763 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:35.079725    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-kube-api-access-5knpz" pod "eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e" (UID: "eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e"). InnerVolumeSpecName "kube-api-access-5knpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:38:35.178778 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:35.178726    4327 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-maas-api-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:38:35.178778 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:35.178770    4327 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-metrics-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:38:35.178778 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:35.178783    4327 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-openshift-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:38:35.179041 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:35.178792    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5knpz\" (UniqueName: \"kubernetes.io/projected/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e-kube-api-access-5knpz\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:38:35.581492 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:35.581455    4327 generic.go:356] "Generic (PLEG): container finished" podID="eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e" containerID="e1f38c9eacef5ac335ba68dc290d768fb6193999eea1f678ba34a59453fbda44" exitCode=0
Oct 31 01:38:35.581673 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:35.581502    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv"
Oct 31 01:38:35.581673 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:35.581539    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv" event={"ID":"eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e","Type":"ContainerDied","Data":"e1f38c9eacef5ac335ba68dc290d768fb6193999eea1f678ba34a59453fbda44"}
Oct 31 01:38:35.581673 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:35.581595    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv" event={"ID":"eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e","Type":"ContainerDied","Data":"41739f3f8bd99c151228bc5568c59a9dbbb73cb2837e7bd9fd28b87bc5df8544"}
Oct 31 01:38:35.581673 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:35.581617    4327 scope.go:122] "RemoveContainer" containerID="e1f38c9eacef5ac335ba68dc290d768fb6193999eea1f678ba34a59453fbda44"
Oct 31 01:38:35.590930 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:35.590914    4327 scope.go:122] "RemoveContainer" containerID="e1f38c9eacef5ac335ba68dc290d768fb6193999eea1f678ba34a59453fbda44"
Oct 31 01:38:35.591202 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:38:35.591182    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1f38c9eacef5ac335ba68dc290d768fb6193999eea1f678ba34a59453fbda44\": container with ID starting with e1f38c9eacef5ac335ba68dc290d768fb6193999eea1f678ba34a59453fbda44 not found: ID does not exist" containerID="e1f38c9eacef5ac335ba68dc290d768fb6193999eea1f678ba34a59453fbda44"
Oct 31 01:38:35.591249 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:35.591212    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1f38c9eacef5ac335ba68dc290d768fb6193999eea1f678ba34a59453fbda44"} err="failed to get container status \"e1f38c9eacef5ac335ba68dc290d768fb6193999eea1f678ba34a59453fbda44\": rpc error: code = NotFound desc = could not find container \"e1f38c9eacef5ac335ba68dc290d768fb6193999eea1f678ba34a59453fbda44\": container with ID starting with e1f38c9eacef5ac335ba68dc290d768fb6193999eea1f678ba34a59453fbda44 not found: ID does not exist"
Oct 31 01:38:35.633846 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:35.633823    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv"]
Oct 31 01:38:35.645830 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:35.645800    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c206c91-7d48865874-plzwv"]
Oct 31 01:38:35.919386 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:35.919353    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e" path="/var/lib/kubelet/pods/eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e/volumes"
Oct 31 01:38:37.690090 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:37.690047    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp" podUID="cc03e2b4-0a09-43ff-8f69-0ba0a590123f" containerName="payload-processing" containerID="cri-o://944cc1b722f9f622a13b72093d649c76a3561feac2b02b9dfb22d78ceeddcc1f" gracePeriod=25
Oct 31 01:38:37.728742 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:37.728702    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8" podUID="0034165d-9b86-4556-806e-f64fc9156adc" containerName="payload-pre-processing" containerID="cri-o://d40984d4fa23459fe67fa5b0b796f44f5096f4d998d3c19daeba021c5659f97b" gracePeriod=25
Oct 31 01:38:37.975466 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:37.975444    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp"
Oct 31 01:38:38.015052 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.015029    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8"
Oct 31 01:38:38.106943 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.106911    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0034165d-9b86-4556-806e-f64fc9156adc-kube-api-access-wc6c8\" (UniqueName: \"kubernetes.io/projected/0034165d-9b86-4556-806e-f64fc9156adc-kube-api-access-wc6c8\") pod \"0034165d-9b86-4556-806e-f64fc9156adc\" (UID: \"0034165d-9b86-4556-806e-f64fc9156adc\") "
Oct 31 01:38:38.106943 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.106947    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc03e2b4-0a09-43ff-8f69-0ba0a590123f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc03e2b4-0a09-43ff-8f69-0ba0a590123f-plugins-config-volume\") pod \"cc03e2b4-0a09-43ff-8f69-0ba0a590123f\" (UID: \"cc03e2b4-0a09-43ff-8f69-0ba0a590123f\") "
Oct 31 01:38:38.107265 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.107000    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc03e2b4-0a09-43ff-8f69-0ba0a590123f-kube-api-access-9dvqc\" (UniqueName: \"kubernetes.io/projected/cc03e2b4-0a09-43ff-8f69-0ba0a590123f-kube-api-access-9dvqc\") pod \"cc03e2b4-0a09-43ff-8f69-0ba0a590123f\" (UID: \"cc03e2b4-0a09-43ff-8f69-0ba0a590123f\") "
Oct 31 01:38:38.107265 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.107036    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0034165d-9b86-4556-806e-f64fc9156adc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0034165d-9b86-4556-806e-f64fc9156adc-plugins-config-volume\") pod \"0034165d-9b86-4556-806e-f64fc9156adc\" (UID: \"0034165d-9b86-4556-806e-f64fc9156adc\") "
Oct 31 01:38:38.107442 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.107418    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc03e2b4-0a09-43ff-8f69-0ba0a590123f-plugins-config-volume" pod "cc03e2b4-0a09-43ff-8f69-0ba0a590123f" (UID: "cc03e2b4-0a09-43ff-8f69-0ba0a590123f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:38:38.107501 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.107421    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0034165d-9b86-4556-806e-f64fc9156adc-plugins-config-volume" pod "0034165d-9b86-4556-806e-f64fc9156adc" (UID: "0034165d-9b86-4556-806e-f64fc9156adc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:38:38.108837 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.108813    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0034165d-9b86-4556-806e-f64fc9156adc-kube-api-access-wc6c8" pod "0034165d-9b86-4556-806e-f64fc9156adc" (UID: "0034165d-9b86-4556-806e-f64fc9156adc"). InnerVolumeSpecName "kube-api-access-wc6c8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:38:38.109027 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.108999    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc03e2b4-0a09-43ff-8f69-0ba0a590123f-kube-api-access-9dvqc" pod "cc03e2b4-0a09-43ff-8f69-0ba0a590123f" (UID: "cc03e2b4-0a09-43ff-8f69-0ba0a590123f"). InnerVolumeSpecName "kube-api-access-9dvqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:38:38.208450 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.208389    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dvqc\" (UniqueName: \"kubernetes.io/projected/cc03e2b4-0a09-43ff-8f69-0ba0a590123f-kube-api-access-9dvqc\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:38:38.208450 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.208415    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0034165d-9b86-4556-806e-f64fc9156adc-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:38:38.208450 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.208425    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc6c8\" (UniqueName: \"kubernetes.io/projected/0034165d-9b86-4556-806e-f64fc9156adc-kube-api-access-wc6c8\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:38:38.208450 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.208433    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc03e2b4-0a09-43ff-8f69-0ba0a590123f-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:38:38.602827 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.602796    4327 generic.go:356] "Generic (PLEG): container finished" podID="0034165d-9b86-4556-806e-f64fc9156adc" containerID="d40984d4fa23459fe67fa5b0b796f44f5096f4d998d3c19daeba021c5659f97b" exitCode=0
Oct 31 01:38:38.603019 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.602860    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8"
Oct 31 01:38:38.603019 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.602888    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8" event={"ID":"0034165d-9b86-4556-806e-f64fc9156adc","Type":"ContainerDied","Data":"d40984d4fa23459fe67fa5b0b796f44f5096f4d998d3c19daeba021c5659f97b"}
Oct 31 01:38:38.603019 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.602958    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8" event={"ID":"0034165d-9b86-4556-806e-f64fc9156adc","Type":"ContainerDied","Data":"aa36212aaaf46ba6f79869fa61a3dd1e606504a2d4ef96b7033db3caefc33746"}
Oct 31 01:38:38.603019 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.602989    4327 scope.go:122] "RemoveContainer" containerID="d40984d4fa23459fe67fa5b0b796f44f5096f4d998d3c19daeba021c5659f97b"
Oct 31 01:38:38.604797 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.604772    4327 generic.go:356] "Generic (PLEG): container finished" podID="cc03e2b4-0a09-43ff-8f69-0ba0a590123f" containerID="944cc1b722f9f622a13b72093d649c76a3561feac2b02b9dfb22d78ceeddcc1f" exitCode=0
Oct 31 01:38:38.604903 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.604854    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp" event={"ID":"cc03e2b4-0a09-43ff-8f69-0ba0a590123f","Type":"ContainerDied","Data":"944cc1b722f9f622a13b72093d649c76a3561feac2b02b9dfb22d78ceeddcc1f"}
Oct 31 01:38:38.604903 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.604896    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp" event={"ID":"cc03e2b4-0a09-43ff-8f69-0ba0a590123f","Type":"ContainerDied","Data":"972b4a7749d365d0067ce755e8113bd2f31b94bcf824dddbd32b1e2916c62636"}
Oct 31 01:38:38.604982 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.604962    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp"
Oct 31 01:38:38.613887 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.613862    4327 scope.go:122] "RemoveContainer" containerID="d40984d4fa23459fe67fa5b0b796f44f5096f4d998d3c19daeba021c5659f97b"
Oct 31 01:38:38.614112 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:38:38.614090    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d40984d4fa23459fe67fa5b0b796f44f5096f4d998d3c19daeba021c5659f97b\": container with ID starting with d40984d4fa23459fe67fa5b0b796f44f5096f4d998d3c19daeba021c5659f97b not found: ID does not exist" containerID="d40984d4fa23459fe67fa5b0b796f44f5096f4d998d3c19daeba021c5659f97b"
Oct 31 01:38:38.614196 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.614124    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d40984d4fa23459fe67fa5b0b796f44f5096f4d998d3c19daeba021c5659f97b"} err="failed to get container status \"d40984d4fa23459fe67fa5b0b796f44f5096f4d998d3c19daeba021c5659f97b\": rpc error: code = NotFound desc = could not find container \"d40984d4fa23459fe67fa5b0b796f44f5096f4d998d3c19daeba021c5659f97b\": container with ID starting with d40984d4fa23459fe67fa5b0b796f44f5096f4d998d3c19daeba021c5659f97b not found: ID does not exist"
Oct 31 01:38:38.614196 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.614146    4327 scope.go:122] "RemoveContainer" containerID="944cc1b722f9f622a13b72093d649c76a3561feac2b02b9dfb22d78ceeddcc1f"
Oct 31 01:38:38.622105 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.622089    4327 scope.go:122] "RemoveContainer" containerID="944cc1b722f9f622a13b72093d649c76a3561feac2b02b9dfb22d78ceeddcc1f"
Oct 31 01:38:38.622343 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:38:38.622328    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"944cc1b722f9f622a13b72093d649c76a3561feac2b02b9dfb22d78ceeddcc1f\": container with ID starting with 944cc1b722f9f622a13b72093d649c76a3561feac2b02b9dfb22d78ceeddcc1f not found: ID does not exist" containerID="944cc1b722f9f622a13b72093d649c76a3561feac2b02b9dfb22d78ceeddcc1f"
Oct 31 01:38:38.622392 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.622350    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"944cc1b722f9f622a13b72093d649c76a3561feac2b02b9dfb22d78ceeddcc1f"} err="failed to get container status \"944cc1b722f9f622a13b72093d649c76a3561feac2b02b9dfb22d78ceeddcc1f\": rpc error: code = NotFound desc = could not find container \"944cc1b722f9f622a13b72093d649c76a3561feac2b02b9dfb22d78ceeddcc1f\": container with ID starting with 944cc1b722f9f622a13b72093d649c76a3561feac2b02b9dfb22d78ceeddcc1f not found: ID does not exist"
Oct 31 01:38:38.655133 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.655112    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8"]
Oct 31 01:38:38.667560 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.667539    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1c206c91-9f7879bdc-n88l8"]
Oct 31 01:38:38.702927 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.702889    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp"]
Oct 31 01:38:38.708956 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:38.708904    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1c206c91-77d45f557c-l6rhp"]
Oct 31 01:38:39.918743 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:39.918714    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0034165d-9b86-4556-806e-f64fc9156adc" path="/var/lib/kubelet/pods/0034165d-9b86-4556-806e-f64fc9156adc/volumes"
Oct 31 01:38:39.919139 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:39.919091    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc03e2b4-0a09-43ff-8f69-0ba0a590123f" path="/var/lib/kubelet/pods/cc03e2b4-0a09-43ff-8f69-0ba0a590123f/volumes"
Oct 31 01:38:40.342616 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.342530    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"]
Oct 31 01:38:40.345095 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.345069    4327 cpu_manager.go:414] "containerMap: removing container" podUID="cc03e2b4-0a09-43ff-8f69-0ba0a590123f" containerName="payload-processing"
Oct 31 01:38:40.345095 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.345093    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc03e2b4-0a09-43ff-8f69-0ba0a590123f" containerName="payload-processing"
Oct 31 01:38:40.345318 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.345108    4327 cpu_manager.go:414] "containerMap: removing container" podUID="0034165d-9b86-4556-806e-f64fc9156adc" containerName="payload-pre-processing"
Oct 31 01:38:40.345318 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.345113    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0034165d-9b86-4556-806e-f64fc9156adc" containerName="payload-pre-processing"
Oct 31 01:38:40.345318 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.345141    4327 cpu_manager.go:414] "containerMap: removing container" podUID="eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e" containerName="maas-api"
Oct 31 01:38:40.345318 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.345147    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e" containerName="maas-api"
Oct 31 01:38:40.345318 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.345243    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc03e2b4-0a09-43ff-8f69-0ba0a590123f" containerName="payload-processing"
Oct 31 01:38:40.345318 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.345252    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="0034165d-9b86-4556-806e-f64fc9156adc" containerName="payload-pre-processing"
Oct 31 01:38:40.345318 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.345262    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb01ca4a-be4f-4fc3-ae20-870e5d0aff0e" containerName="maas-api"
Oct 31 01:38:40.349971 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.349952    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.367981 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.367957    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-aebdfd-openshift-default-dockercfg-qxhct\""
Oct 31 01:38:40.382930 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.382903    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"]
Oct 31 01:38:40.429652 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.429623    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xp7l\" (UniqueName: \"kubernetes.io/projected/6b74fa47-62f0-44e3-b56c-a52497d85960-kube-api-access-6xp7l\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.429851 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.429663    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-workload-certs\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.429851 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.429768    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-token\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.429851 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.429797    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-workload-socket\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.429851 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.429817    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b74fa47-62f0-44e3-b56c-a52497d85960-istiod-ca-cert\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.429851 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.429835    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-podinfo\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.430131 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.429857    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-data\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.430131 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.429914    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-credential-socket\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.430131 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.430027    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-envoy\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.531198 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.531155    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xp7l\" (UniqueName: \"kubernetes.io/projected/6b74fa47-62f0-44e3-b56c-a52497d85960-kube-api-access-6xp7l\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.531419 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.531213    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-workload-certs\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.531419 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.531249    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-token\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.531419 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.531276    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-workload-socket\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.531419 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.531304    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b74fa47-62f0-44e3-b56c-a52497d85960-istiod-ca-cert\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.531419 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.531329    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-podinfo\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.531419 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.531362    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-data\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.531419 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.531388    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-credential-socket\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.531819 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.531439    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-envoy\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.531819 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.531740    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-workload-certs\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.531819 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.531764    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-workload-socket\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.532279 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.532250    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b74fa47-62f0-44e3-b56c-a52497d85960-istiod-ca-cert\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.532394 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.532371    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-credential-socket\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.532464 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.532418    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-data\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.533840 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.533820    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-envoy\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.534285 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.534267    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-podinfo\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.557211 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.557183    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xp7l\" (UniqueName: \"kubernetes.io/projected/6b74fa47-62f0-44e3-b56c-a52497d85960-kube-api-access-6xp7l\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.557342 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.557273    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-token\") pod \"e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") " pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.674060 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.674019    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:40.985320 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.985290    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"]
Oct 31 01:38:40.987917 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:38:40.987882    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b74fa47_62f0_44e3_b56c_a52497d85960.slice/crio-e8d193f402d49820788f442952b1ede825c8ad3688ec069a8c8115eb0f61c302 WatchSource:0}: Error finding container e8d193f402d49820788f442952b1ede825c8ad3688ec069a8c8115eb0f61c302: Status 404 returned error can't find the container with id e8d193f402d49820788f442952b1ede825c8ad3688ec069a8c8115eb0f61c302
Oct 31 01:38:40.990084 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.990050    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:38:40.990171 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.990117    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:38:40.990171 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:40.990144    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:38:41.629726 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:41.629689    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt" event={"ID":"6b74fa47-62f0-44e3-b56c-a52497d85960","Type":"ContainerStarted","Data":"de665c4ec0ceef00c07111c60d90afac457b672f95f2ce7d5eeb174ff8a87203"}
Oct 31 01:38:41.629936 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:41.629762    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt" event={"ID":"6b74fa47-62f0-44e3-b56c-a52497d85960","Type":"ContainerStarted","Data":"e8d193f402d49820788f442952b1ede825c8ad3688ec069a8c8115eb0f61c302"}
Oct 31 01:38:41.674107 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:41.674076    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:41.677042 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:41.676995    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt" podStartSLOduration=1.6769819099999999 podStartE2EDuration="1.67698191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:41.674318443 +0000 UTC m=+4781.896353997" watchObservedRunningTime="2026-10-31 01:38:41.67698191 +0000 UTC m=+4781.899017452"
Oct 31 01:38:42.679357 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:42.679323    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:43.637521 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:43.637486    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:43.638497 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:43.638470    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:38:45.292348 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:38:45.292305    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:38:45.890431 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:45.890397    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz"]
Oct 31 01:38:45.894011 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:45.893988    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz"
Oct 31 01:38:45.908835 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:45.908816    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-aebdfd\""
Oct 31 01:38:45.922896 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:45.922871    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz"]
Oct 31 01:38:45.959603 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:45.959567    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s"]
Oct 31 01:38:45.963136 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:45.963115    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s"
Oct 31 01:38:45.984818 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:45.984801    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-aebdfd-dockercfg-qscgh\""
Oct 31 01:38:45.987211 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:45.987191    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35b06a6a-0926-42de-8a8c-ba6aa58ef48a-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz\" (UID: \"35b06a6a-0926-42de-8a8c-ba6aa58ef48a\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz"
Oct 31 01:38:45.987345 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:45.987330    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9jbk\" (UniqueName: \"kubernetes.io/projected/35b06a6a-0926-42de-8a8c-ba6aa58ef48a-kube-api-access-s9jbk\") pod \"payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz\" (UID: \"35b06a6a-0926-42de-8a8c-ba6aa58ef48a\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz"
Oct 31 01:38:45.992486 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:45.992451    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s"]
Oct 31 01:38:46.088288 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:46.088254    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9hf9\" (UniqueName: \"kubernetes.io/projected/8d65cd0c-49f1-4b1b-8872-6693bc6c982e-kube-api-access-d9hf9\") pod \"payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s\" (UID: \"8d65cd0c-49f1-4b1b-8872-6693bc6c982e\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s"
Oct 31 01:38:46.088500 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:46.088312    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9jbk\" (UniqueName: \"kubernetes.io/projected/35b06a6a-0926-42de-8a8c-ba6aa58ef48a-kube-api-access-s9jbk\") pod \"payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz\" (UID: \"35b06a6a-0926-42de-8a8c-ba6aa58ef48a\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz"
Oct 31 01:38:46.088500 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:46.088460    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35b06a6a-0926-42de-8a8c-ba6aa58ef48a-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz\" (UID: \"35b06a6a-0926-42de-8a8c-ba6aa58ef48a\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz"
Oct 31 01:38:46.088649 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:46.088505    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d65cd0c-49f1-4b1b-8872-6693bc6c982e-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s\" (UID: \"8d65cd0c-49f1-4b1b-8872-6693bc6c982e\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s"
Oct 31 01:38:46.089302 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:46.089284    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35b06a6a-0926-42de-8a8c-ba6aa58ef48a-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz\" (UID: \"35b06a6a-0926-42de-8a8c-ba6aa58ef48a\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz"
Oct 31 01:38:46.115180 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:46.115151    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9jbk\" (UniqueName: \"kubernetes.io/projected/35b06a6a-0926-42de-8a8c-ba6aa58ef48a-kube-api-access-s9jbk\") pod \"payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz\" (UID: \"35b06a6a-0926-42de-8a8c-ba6aa58ef48a\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz"
Oct 31 01:38:46.190085 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:46.189995    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d65cd0c-49f1-4b1b-8872-6693bc6c982e-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s\" (UID: \"8d65cd0c-49f1-4b1b-8872-6693bc6c982e\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s"
Oct 31 01:38:46.190085 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:46.190061    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9hf9\" (UniqueName: \"kubernetes.io/projected/8d65cd0c-49f1-4b1b-8872-6693bc6c982e-kube-api-access-d9hf9\") pod \"payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s\" (UID: \"8d65cd0c-49f1-4b1b-8872-6693bc6c982e\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s"
Oct 31 01:38:46.190734 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:46.190711    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d65cd0c-49f1-4b1b-8872-6693bc6c982e-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s\" (UID: \"8d65cd0c-49f1-4b1b-8872-6693bc6c982e\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s"
Oct 31 01:38:46.207840 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:46.207812    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz"
Oct 31 01:38:46.221032 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:46.221003    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9hf9\" (UniqueName: \"kubernetes.io/projected/8d65cd0c-49f1-4b1b-8872-6693bc6c982e-kube-api-access-d9hf9\") pod \"payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s\" (UID: \"8d65cd0c-49f1-4b1b-8872-6693bc6c982e\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s"
Oct 31 01:38:46.276518 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:46.276165    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s"
Oct 31 01:38:46.506076 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:46.506051    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz"]
Oct 31 01:38:46.508709 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:38:46.508666    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b06a6a_0926_42de_8a8c_ba6aa58ef48a.slice/crio-09198d047f2796de1867e4fe03fc1956d697dd1ed0a2e409e8963d08f4b00b4e WatchSource:0}: Error finding container 09198d047f2796de1867e4fe03fc1956d697dd1ed0a2e409e8963d08f4b00b4e: Status 404 returned error can't find the container with id 09198d047f2796de1867e4fe03fc1956d697dd1ed0a2e409e8963d08f4b00b4e
Oct 31 01:38:46.582829 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:46.582809    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s"]
Oct 31 01:38:46.585025 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:38:46.585000    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d65cd0c_49f1_4b1b_8872_6693bc6c982e.slice/crio-0e93d5688d321f4e8e7746936024f9301bce15ffa1e53b606f9a4992aa1923f6 WatchSource:0}: Error finding container 0e93d5688d321f4e8e7746936024f9301bce15ffa1e53b606f9a4992aa1923f6: Status 404 returned error can't find the container with id 0e93d5688d321f4e8e7746936024f9301bce15ffa1e53b606f9a4992aa1923f6
Oct 31 01:38:46.652966 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:46.652932    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s" event={"ID":"8d65cd0c-49f1-4b1b-8872-6693bc6c982e","Type":"ContainerStarted","Data":"0e93d5688d321f4e8e7746936024f9301bce15ffa1e53b606f9a4992aa1923f6"}
Oct 31 01:38:46.654946 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:46.654768    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz" event={"ID":"35b06a6a-0926-42de-8a8c-ba6aa58ef48a","Type":"ContainerStarted","Data":"3e32eaea67ecfc3adf4db89904172e8b8f201f612b413d34211095a3a5454784"}
Oct 31 01:38:46.654946 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:46.654825    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz" event={"ID":"35b06a6a-0926-42de-8a8c-ba6aa58ef48a","Type":"ContainerStarted","Data":"09198d047f2796de1867e4fe03fc1956d697dd1ed0a2e409e8963d08f4b00b4e"}
Oct 31 01:38:46.655085 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:46.655002    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz"
Oct 31 01:38:46.701022 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:46.700969    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz" podStartSLOduration=1.700955178 podStartE2EDuration="1.700955178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:46.698585008 +0000 UTC m=+4786.920620550" watchObservedRunningTime="2026-10-31 01:38:46.700955178 +0000 UTC m=+4786.922990719"
Oct 31 01:38:47.661020 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:47.660978    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s" event={"ID":"8d65cd0c-49f1-4b1b-8872-6693bc6c982e","Type":"ContainerStarted","Data":"db96e1dce66884c8e5fcd767fed36f2ad9adf0967e926d189548dddd7af297a2"}
Oct 31 01:38:47.661509 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:47.661049    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s"
Oct 31 01:38:47.701369 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:47.701311    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s" podStartSLOduration=2.701296835 podStartE2EDuration="2.701296835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:47.700587283 +0000 UTC m=+4787.922622823" watchObservedRunningTime="2026-10-31 01:38:47.701296835 +0000 UTC m=+4787.923332378"
Oct 31 01:38:57.664436 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:57.664404    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz"
Oct 31 01:38:58.639152 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:58.639112    4327 patch_prober.go:29] interesting pod/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 31 01:38:58.639395 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:58.639181    4327 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt" podUID="6b74fa47-62f0-44e3-b56c-a52497d85960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 31 01:38:58.668907 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:58.668816    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s"
Oct 31 01:38:58.716531 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:58.716492    4327 generic.go:356] "Generic (PLEG): container finished" podID="6b74fa47-62f0-44e3-b56c-a52497d85960" containerID="de665c4ec0ceef00c07111c60d90afac457b672f95f2ce7d5eeb174ff8a87203" exitCode=0
Oct 31 01:38:58.716739 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:58.716569    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt" event={"ID":"6b74fa47-62f0-44e3-b56c-a52497d85960","Type":"ContainerDied","Data":"de665c4ec0ceef00c07111c60d90afac457b672f95f2ce7d5eeb174ff8a87203"}
Oct 31 01:38:58.717090 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:58.717071    4327 scope.go:122] "RemoveContainer" containerID="de665c4ec0ceef00c07111c60d90afac457b672f95f2ce7d5eeb174ff8a87203"
Oct 31 01:38:58.719172 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:58.719138    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:38:58.719281 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:58.719224    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:38:58.719350 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:58.719277    4327 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 31 01:38:59.745710 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:38:59.745647    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt" event={"ID":"6b74fa47-62f0-44e3-b56c-a52497d85960","Type":"ContainerStarted","Data":"3b66c4aa300caa98fc99c0b587583df42d1af1380c3aed1a29732cf446cc32f0"}
Oct 31 01:39:00.674165 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:39:00.674131    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:39:00.674408 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:39:00.674383    4327 patch_prober.go:29] interesting pod/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 31 01:39:00.674481 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:39:00.674463    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt" podUID="6b74fa47-62f0-44e3-b56c-a52497d85960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 31 01:39:01.675037 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:39:01.674999    4327 patch_prober.go:29] interesting pod/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 31 01:39:01.675535 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:39:01.675064    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt" podUID="6b74fa47-62f0-44e3-b56c-a52497d85960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 31 01:39:02.674492 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:39:02.674447    4327 patch_prober.go:29] interesting pod/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 31 01:39:02.674730 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:39:02.674514    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt" podUID="6b74fa47-62f0-44e3-b56c-a52497d85960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 31 01:39:03.675061 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:39:03.675023    4327 patch_prober.go:29] interesting pod/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 31 01:39:03.675807 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:39:03.675076    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt" podUID="6b74fa47-62f0-44e3-b56c-a52497d85960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 31 01:39:04.675113 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:39:04.675073    4327 patch_prober.go:29] interesting pod/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 31 01:39:04.675515 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:39:04.675133    4327 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt" podUID="6b74fa47-62f0-44e3-b56c-a52497d85960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 31 01:39:05.678558 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:39:05.678528    4327 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:39:05.679065 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:39:05.679002    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:39:05.679755 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:39:05.679732    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:39:15.292060 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:39:15.291972    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:39:45.291470 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:39:45.291432    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:40:15.292564 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:40:15.292527    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:40:29.791771 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:29.791733    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz"]
Oct 31 01:40:29.824189 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:29.824157    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s"]
Oct 31 01:40:34.792279 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:34.792216    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz" podUID="35b06a6a-0926-42de-8a8c-ba6aa58ef48a" containerName="payload-processing" containerID="cri-o://3e32eaea67ecfc3adf4db89904172e8b8f201f612b413d34211095a3a5454784" gracePeriod=25
Oct 31 01:40:34.824467 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:34.824432    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s" podUID="8d65cd0c-49f1-4b1b-8872-6693bc6c982e" containerName="payload-pre-processing" containerID="cri-o://db96e1dce66884c8e5fcd767fed36f2ad9adf0967e926d189548dddd7af297a2" gracePeriod=25
Oct 31 01:40:35.072337 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.072311    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz"
Oct 31 01:40:35.101542 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.101517    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s"
Oct 31 01:40:35.193413 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.193379    4327 generic.go:356] "Generic (PLEG): container finished" podID="8d65cd0c-49f1-4b1b-8872-6693bc6c982e" containerID="db96e1dce66884c8e5fcd767fed36f2ad9adf0967e926d189548dddd7af297a2" exitCode=0
Oct 31 01:40:35.193613 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.193423    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s"
Oct 31 01:40:35.193613 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.193465    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s" event={"ID":"8d65cd0c-49f1-4b1b-8872-6693bc6c982e","Type":"ContainerDied","Data":"db96e1dce66884c8e5fcd767fed36f2ad9adf0967e926d189548dddd7af297a2"}
Oct 31 01:40:35.193613 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.193529    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s" event={"ID":"8d65cd0c-49f1-4b1b-8872-6693bc6c982e","Type":"ContainerDied","Data":"0e93d5688d321f4e8e7746936024f9301bce15ffa1e53b606f9a4992aa1923f6"}
Oct 31 01:40:35.193613 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.193564    4327 scope.go:122] "RemoveContainer" containerID="db96e1dce66884c8e5fcd767fed36f2ad9adf0967e926d189548dddd7af297a2"
Oct 31 01:40:35.194785 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.194741    4327 generic.go:356] "Generic (PLEG): container finished" podID="35b06a6a-0926-42de-8a8c-ba6aa58ef48a" containerID="3e32eaea67ecfc3adf4db89904172e8b8f201f612b413d34211095a3a5454784" exitCode=0
Oct 31 01:40:35.194854 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.194786    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz" event={"ID":"35b06a6a-0926-42de-8a8c-ba6aa58ef48a","Type":"ContainerDied","Data":"3e32eaea67ecfc3adf4db89904172e8b8f201f612b413d34211095a3a5454784"}
Oct 31 01:40:35.194854 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.194802    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz"
Oct 31 01:40:35.194854 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.194825    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz" event={"ID":"35b06a6a-0926-42de-8a8c-ba6aa58ef48a","Type":"ContainerDied","Data":"09198d047f2796de1867e4fe03fc1956d697dd1ed0a2e409e8963d08f4b00b4e"}
Oct 31 01:40:35.200000 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.199978    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35b06a6a-0926-42de-8a8c-ba6aa58ef48a-kube-api-access-s9jbk\" (UniqueName: \"kubernetes.io/projected/35b06a6a-0926-42de-8a8c-ba6aa58ef48a-kube-api-access-s9jbk\") pod \"35b06a6a-0926-42de-8a8c-ba6aa58ef48a\" (UID: \"35b06a6a-0926-42de-8a8c-ba6aa58ef48a\") "
Oct 31 01:40:35.200126 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.200110    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d65cd0c-49f1-4b1b-8872-6693bc6c982e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d65cd0c-49f1-4b1b-8872-6693bc6c982e-plugins-config-volume\") pod \"8d65cd0c-49f1-4b1b-8872-6693bc6c982e\" (UID: \"8d65cd0c-49f1-4b1b-8872-6693bc6c982e\") "
Oct 31 01:40:35.200178 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.200165    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35b06a6a-0926-42de-8a8c-ba6aa58ef48a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35b06a6a-0926-42de-8a8c-ba6aa58ef48a-plugins-config-volume\") pod \"35b06a6a-0926-42de-8a8c-ba6aa58ef48a\" (UID: \"35b06a6a-0926-42de-8a8c-ba6aa58ef48a\") "
Oct 31 01:40:35.200224 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.200204    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d65cd0c-49f1-4b1b-8872-6693bc6c982e-kube-api-access-d9hf9\" (UniqueName: \"kubernetes.io/projected/8d65cd0c-49f1-4b1b-8872-6693bc6c982e-kube-api-access-d9hf9\") pod \"8d65cd0c-49f1-4b1b-8872-6693bc6c982e\" (UID: \"8d65cd0c-49f1-4b1b-8872-6693bc6c982e\") "
Oct 31 01:40:35.200486 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.200466    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d65cd0c-49f1-4b1b-8872-6693bc6c982e-plugins-config-volume" pod "8d65cd0c-49f1-4b1b-8872-6693bc6c982e" (UID: "8d65cd0c-49f1-4b1b-8872-6693bc6c982e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:35.200568 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.200522    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b06a6a-0926-42de-8a8c-ba6aa58ef48a-plugins-config-volume" pod "35b06a6a-0926-42de-8a8c-ba6aa58ef48a" (UID: "35b06a6a-0926-42de-8a8c-ba6aa58ef48a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:35.200617 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.200588    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d65cd0c-49f1-4b1b-8872-6693bc6c982e-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:40:35.202027 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.202003    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b06a6a-0926-42de-8a8c-ba6aa58ef48a-kube-api-access-s9jbk" pod "35b06a6a-0926-42de-8a8c-ba6aa58ef48a" (UID: "35b06a6a-0926-42de-8a8c-ba6aa58ef48a"). InnerVolumeSpecName "kube-api-access-s9jbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:35.202160 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.202144    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d65cd0c-49f1-4b1b-8872-6693bc6c982e-kube-api-access-d9hf9" pod "8d65cd0c-49f1-4b1b-8872-6693bc6c982e" (UID: "8d65cd0c-49f1-4b1b-8872-6693bc6c982e"). InnerVolumeSpecName "kube-api-access-d9hf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:35.202640 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.202624    4327 scope.go:122] "RemoveContainer" containerID="db96e1dce66884c8e5fcd767fed36f2ad9adf0967e926d189548dddd7af297a2"
Oct 31 01:40:35.202911 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:40:35.202891    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db96e1dce66884c8e5fcd767fed36f2ad9adf0967e926d189548dddd7af297a2\": container with ID starting with db96e1dce66884c8e5fcd767fed36f2ad9adf0967e926d189548dddd7af297a2 not found: ID does not exist" containerID="db96e1dce66884c8e5fcd767fed36f2ad9adf0967e926d189548dddd7af297a2"
Oct 31 01:40:35.202973 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.202919    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db96e1dce66884c8e5fcd767fed36f2ad9adf0967e926d189548dddd7af297a2"} err="failed to get container status \"db96e1dce66884c8e5fcd767fed36f2ad9adf0967e926d189548dddd7af297a2\": rpc error: code = NotFound desc = could not find container \"db96e1dce66884c8e5fcd767fed36f2ad9adf0967e926d189548dddd7af297a2\": container with ID starting with db96e1dce66884c8e5fcd767fed36f2ad9adf0967e926d189548dddd7af297a2 not found: ID does not exist"
Oct 31 01:40:35.202973 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.202935    4327 scope.go:122] "RemoveContainer" containerID="3e32eaea67ecfc3adf4db89904172e8b8f201f612b413d34211095a3a5454784"
Oct 31 01:40:35.210387 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.210371    4327 scope.go:122] "RemoveContainer" containerID="3e32eaea67ecfc3adf4db89904172e8b8f201f612b413d34211095a3a5454784"
Oct 31 01:40:35.210670 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:40:35.210651    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e32eaea67ecfc3adf4db89904172e8b8f201f612b413d34211095a3a5454784\": container with ID starting with 3e32eaea67ecfc3adf4db89904172e8b8f201f612b413d34211095a3a5454784 not found: ID does not exist" containerID="3e32eaea67ecfc3adf4db89904172e8b8f201f612b413d34211095a3a5454784"
Oct 31 01:40:35.210741 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.210691    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e32eaea67ecfc3adf4db89904172e8b8f201f612b413d34211095a3a5454784"} err="failed to get container status \"3e32eaea67ecfc3adf4db89904172e8b8f201f612b413d34211095a3a5454784\": rpc error: code = NotFound desc = could not find container \"3e32eaea67ecfc3adf4db89904172e8b8f201f612b413d34211095a3a5454784\": container with ID starting with 3e32eaea67ecfc3adf4db89904172e8b8f201f612b413d34211095a3a5454784 not found: ID does not exist"
Oct 31 01:40:35.302085 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.302005    4327 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35b06a6a-0926-42de-8a8c-ba6aa58ef48a-plugins-config-volume\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:40:35.302085 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.302037    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9hf9\" (UniqueName: \"kubernetes.io/projected/8d65cd0c-49f1-4b1b-8872-6693bc6c982e-kube-api-access-d9hf9\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:40:35.302085 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.302047    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9jbk\" (UniqueName: \"kubernetes.io/projected/35b06a6a-0926-42de-8a8c-ba6aa58ef48a-kube-api-access-s9jbk\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:40:35.543533 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.543499    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz"]
Oct 31 01:40:35.555962 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.555904    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-aebdfd-c6f7cc4bc-pf5zz"]
Oct 31 01:40:35.581731 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.581705    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s"]
Oct 31 01:40:35.586048 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.586027    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-aebdfd-7688679755-tp78s"]
Oct 31 01:40:35.922080 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.922041    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35b06a6a-0926-42de-8a8c-ba6aa58ef48a" path="/var/lib/kubelet/pods/35b06a6a-0926-42de-8a8c-ba6aa58ef48a/volumes"
Oct 31 01:40:35.922668 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:35.922648    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d65cd0c-49f1-4b1b-8872-6693bc6c982e" path="/var/lib/kubelet/pods/8d65cd0c-49f1-4b1b-8872-6693bc6c982e/volumes"
Oct 31 01:40:43.461955 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:43.461855    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"]
Oct 31 01:40:43.462478 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:43.462131    4327 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt" podUID="6b74fa47-62f0-44e3-b56c-a52497d85960" containerName="istio-proxy" containerID="cri-o://3b66c4aa300caa98fc99c0b587583df42d1af1380c3aed1a29732cf446cc32f0" gracePeriod=30
Oct 31 01:40:45.290854 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:40:45.290810    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:40:48.914035 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:48.914005    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:40:49.083230 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.083195    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-data\") pod \"6b74fa47-62f0-44e3-b56c-a52497d85960\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") "
Oct 31 01:40:49.083430 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.083262    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-workload-certs\") pod \"6b74fa47-62f0-44e3-b56c-a52497d85960\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") "
Oct 31 01:40:49.083430 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.083293    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b74fa47-62f0-44e3-b56c-a52497d85960-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b74fa47-62f0-44e3-b56c-a52497d85960-istiod-ca-cert\") pod \"6b74fa47-62f0-44e3-b56c-a52497d85960\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") "
Oct 31 01:40:49.083430 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.083338    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-workload-socket\") pod \"6b74fa47-62f0-44e3-b56c-a52497d85960\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") "
Oct 31 01:40:49.083430 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.083390    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b74fa47-62f0-44e3-b56c-a52497d85960-kube-api-access-6xp7l\" (UniqueName: \"kubernetes.io/projected/6b74fa47-62f0-44e3-b56c-a52497d85960-kube-api-access-6xp7l\") pod \"6b74fa47-62f0-44e3-b56c-a52497d85960\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") "
Oct 31 01:40:49.083763 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.083484    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-podinfo\") pod \"6b74fa47-62f0-44e3-b56c-a52497d85960\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") "
Oct 31 01:40:49.083763 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.083491    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-data" pod "6b74fa47-62f0-44e3-b56c-a52497d85960" (UID: "6b74fa47-62f0-44e3-b56c-a52497d85960"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:49.083763 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.083531    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-credential-socket\") pod \"6b74fa47-62f0-44e3-b56c-a52497d85960\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") "
Oct 31 01:40:49.083763 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.083564    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-envoy\") pod \"6b74fa47-62f0-44e3-b56c-a52497d85960\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") "
Oct 31 01:40:49.083763 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.083530    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-workload-certs" pod "6b74fa47-62f0-44e3-b56c-a52497d85960" (UID: "6b74fa47-62f0-44e3-b56c-a52497d85960"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:49.083763 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.083637    4327 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-token\" (UniqueName: \"kubernetes.io/projected/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-token\") pod \"6b74fa47-62f0-44e3-b56c-a52497d85960\" (UID: \"6b74fa47-62f0-44e3-b56c-a52497d85960\") "
Oct 31 01:40:49.083763 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.083661    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-workload-socket" pod "6b74fa47-62f0-44e3-b56c-a52497d85960" (UID: "6b74fa47-62f0-44e3-b56c-a52497d85960"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:49.084131 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.083781    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-credential-socket" pod "6b74fa47-62f0-44e3-b56c-a52497d85960" (UID: "6b74fa47-62f0-44e3-b56c-a52497d85960"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:49.084131 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.083788    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b74fa47-62f0-44e3-b56c-a52497d85960-istiod-ca-cert" pod "6b74fa47-62f0-44e3-b56c-a52497d85960" (UID: "6b74fa47-62f0-44e3-b56c-a52497d85960"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:49.084131 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.084006    4327 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-credential-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:40:49.084131 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.084027    4327 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-data\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:40:49.084131 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.084041    4327 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-workload-certs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:40:49.084131 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.084054    4327 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b74fa47-62f0-44e3-b56c-a52497d85960-istiod-ca-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:40:49.084131 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.084067    4327 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-workload-socket\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:40:49.085759 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.085729    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-podinfo" pod "6b74fa47-62f0-44e3-b56c-a52497d85960" (UID: "6b74fa47-62f0-44e3-b56c-a52497d85960"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:40:49.085759 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.085744    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-envoy" pod "6b74fa47-62f0-44e3-b56c-a52497d85960" (UID: "6b74fa47-62f0-44e3-b56c-a52497d85960"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:49.086041 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.086017    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-token" pod "6b74fa47-62f0-44e3-b56c-a52497d85960" (UID: "6b74fa47-62f0-44e3-b56c-a52497d85960"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:49.086096 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.086048    4327 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b74fa47-62f0-44e3-b56c-a52497d85960-kube-api-access-6xp7l" pod "6b74fa47-62f0-44e3-b56c-a52497d85960" (UID: "6b74fa47-62f0-44e3-b56c-a52497d85960"). InnerVolumeSpecName "kube-api-access-6xp7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:49.185110 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.185045    4327 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-token\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:40:49.185110 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.185070    4327 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xp7l\" (UniqueName: \"kubernetes.io/projected/6b74fa47-62f0-44e3-b56c-a52497d85960-kube-api-access-6xp7l\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:40:49.185110 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.185080    4327 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-podinfo\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:40:49.185110 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.185089    4327 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b74fa47-62f0-44e3-b56c-a52497d85960-istio-envoy\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Oct 31 01:40:49.295897 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.295860    4327 generic.go:356] "Generic (PLEG): container finished" podID="6b74fa47-62f0-44e3-b56c-a52497d85960" containerID="3b66c4aa300caa98fc99c0b587583df42d1af1380c3aed1a29732cf446cc32f0" exitCode=0
Oct 31 01:40:49.296078 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.295907    4327 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"
Oct 31 01:40:49.296078 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.295939    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt" event={"ID":"6b74fa47-62f0-44e3-b56c-a52497d85960","Type":"ContainerDied","Data":"3b66c4aa300caa98fc99c0b587583df42d1af1380c3aed1a29732cf446cc32f0"}
Oct 31 01:40:49.296078 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.295992    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt" event={"ID":"6b74fa47-62f0-44e3-b56c-a52497d85960","Type":"ContainerDied","Data":"e8d193f402d49820788f442952b1ede825c8ad3688ec069a8c8115eb0f61c302"}
Oct 31 01:40:49.296078 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.296017    4327 scope.go:122] "RemoveContainer" containerID="3b66c4aa300caa98fc99c0b587583df42d1af1380c3aed1a29732cf446cc32f0"
Oct 31 01:40:49.305404 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.305382    4327 scope.go:122] "RemoveContainer" containerID="de665c4ec0ceef00c07111c60d90afac457b672f95f2ce7d5eeb174ff8a87203"
Oct 31 01:40:49.313369 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.313353    4327 scope.go:122] "RemoveContainer" containerID="3b66c4aa300caa98fc99c0b587583df42d1af1380c3aed1a29732cf446cc32f0"
Oct 31 01:40:49.313610 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:40:49.313588    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b66c4aa300caa98fc99c0b587583df42d1af1380c3aed1a29732cf446cc32f0\": container with ID starting with 3b66c4aa300caa98fc99c0b587583df42d1af1380c3aed1a29732cf446cc32f0 not found: ID does not exist" containerID="3b66c4aa300caa98fc99c0b587583df42d1af1380c3aed1a29732cf446cc32f0"
Oct 31 01:40:49.313723 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.313616    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b66c4aa300caa98fc99c0b587583df42d1af1380c3aed1a29732cf446cc32f0"} err="failed to get container status \"3b66c4aa300caa98fc99c0b587583df42d1af1380c3aed1a29732cf446cc32f0\": rpc error: code = NotFound desc = could not find container \"3b66c4aa300caa98fc99c0b587583df42d1af1380c3aed1a29732cf446cc32f0\": container with ID starting with 3b66c4aa300caa98fc99c0b587583df42d1af1380c3aed1a29732cf446cc32f0 not found: ID does not exist"
Oct 31 01:40:49.313723 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.313632    4327 scope.go:122] "RemoveContainer" containerID="de665c4ec0ceef00c07111c60d90afac457b672f95f2ce7d5eeb174ff8a87203"
Oct 31 01:40:49.313887 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:40:49.313867    4327 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de665c4ec0ceef00c07111c60d90afac457b672f95f2ce7d5eeb174ff8a87203\": container with ID starting with de665c4ec0ceef00c07111c60d90afac457b672f95f2ce7d5eeb174ff8a87203 not found: ID does not exist" containerID="de665c4ec0ceef00c07111c60d90afac457b672f95f2ce7d5eeb174ff8a87203"
Oct 31 01:40:49.313932 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.313893    4327 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de665c4ec0ceef00c07111c60d90afac457b672f95f2ce7d5eeb174ff8a87203"} err="failed to get container status \"de665c4ec0ceef00c07111c60d90afac457b672f95f2ce7d5eeb174ff8a87203\": rpc error: code = NotFound desc = could not find container \"de665c4ec0ceef00c07111c60d90afac457b672f95f2ce7d5eeb174ff8a87203\": container with ID starting with de665c4ec0ceef00c07111c60d90afac457b672f95f2ce7d5eeb174ff8a87203 not found: ID does not exist"
Oct 31 01:40:49.348600 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.348574    4327 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"]
Oct 31 01:40:49.360599 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.360581    4327 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-aebdfd-openshift-default-7d567cc99c-8gdrt"]
Oct 31 01:40:49.920283 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:40:49.920253    4327 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b74fa47-62f0-44e3-b56c-a52497d85960" path="/var/lib/kubelet/pods/6b74fa47-62f0-44e3-b56c-a52497d85960/volumes"
Oct 31 01:41:15.297356 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:41:15.297306    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:41:45.291544 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:41:45.291497    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:41:51.973626 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:41:51.973598    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-79889fd95f-btjgl_5cfa01dd-f048-4498-ba79-629852aaff37/manager/0.log"
Oct 31 01:41:52.992127 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:41:52.992097    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9c8d8b4d5-z5sc9_0121470e-c1cd-459c-ba8b-0c6f8cac9c4f/manager/0.log"
Oct 31 01:41:54.470641 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:41:54.470610    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-749d798f87-t7cvd_900c9d95-ccc6-4663-b994-2c4a2d3188c6/maas-api/0.log"
Oct 31 01:41:56.000287 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:41:56.000252    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-24d65_c50004be-2445-40ac-a248-ef5d679f9792/postgres/0.log"
Oct 31 01:41:58.175764 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:41:58.175732    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-k7vdf_e0686e3b-1c7a-42f1-b7be-7c85553be299/kuadrant-console-plugin/0.log"
Oct 31 01:41:59.897589 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:41:59.897553    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-frj8v_97aa8115-64f5-478f-93b1-276dc60acd6a/discovery/0.log"
Oct 31 01:42:00.101393 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:42:00.101365    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-94bcdf98f-g9zcg_3abd5af3-5337-4948-849b-b6d2976f269e/kube-auth-proxy/0.log"
Oct 31 01:42:01.060468 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:42:01.060435    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-hlnsl_97c56d5c-fa82-4ff9-b039-dd2839271bc7/istio-proxy/0.log"
Oct 31 01:42:01.464952 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:42:01.464922    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-mbzdk_738f0544-ef43-4c6c-a9ec-a20a362c482d/istio-proxy/0.log"
Oct 31 01:42:02.295169 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:42:02.295133    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-zrjvf_9b1dad6d-c028-4673-96f1-48c4fca807c4/istio-proxy/0.log"
Oct 31 01:42:02.682695 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:42:02.682650    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-64nxw_852f9567-a315-448d-b286-f90afbcaad43/payload-pre-processing/0.log"
Oct 31 01:42:04.089861 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:42:04.089833    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj_cad8e534-8589-407a-bbcf-0bb145b80233/main/0.log"
Oct 31 01:42:04.713605 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:42:04.713571    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-s24sn_e2f2f0dd-a9a0-4335-9c33-683562fa8fbe/main/0.log"
Oct 31 01:42:04.921550 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:42:04.921517    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q_7c1e66f4-2552-43b1-96e0-272661898154/main/0.log"
Oct 31 01:42:05.139158 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:42:05.139130    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dddkj_b1988c46-0b7f-4edd-b441-ab408ad0a246/main/0.log"
Oct 31 01:42:05.324339 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:42:05.324300    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt_c1398a6a-1ac9-48e1-a988-19c575e0cb91/main/0.log"
Oct 31 01:42:15.295047 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:42:15.294960    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:42:15.770007 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:42:15.769972    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-79889fd95f-btjgl_5cfa01dd-f048-4498-ba79-629852aaff37/manager/0.log"
Oct 31 01:42:16.688046 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:42:16.688018    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9c8d8b4d5-z5sc9_0121470e-c1cd-459c-ba8b-0c6f8cac9c4f/manager/0.log"
Oct 31 01:42:17.914836 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:42:17.914812    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-749d798f87-t7cvd_900c9d95-ccc6-4663-b994-2c4a2d3188c6/maas-api/0.log"
Oct 31 01:42:19.278984 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:42:19.278950    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-24d65_c50004be-2445-40ac-a248-ef5d679f9792/postgres/0.log"
Oct 31 01:42:21.049697 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:42:21.049652    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-k7vdf_e0686e3b-1c7a-42f1-b7be-7c85553be299/kuadrant-console-plugin/0.log"
Oct 31 01:42:22.468125 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:42:22.468093    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-frj8v_97aa8115-64f5-478f-93b1-276dc60acd6a/discovery/0.log"
Oct 31 01:42:22.683630 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:42:22.683604    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-94bcdf98f-g9zcg_3abd5af3-5337-4948-849b-b6d2976f269e/kube-auth-proxy/0.log"
Oct 31 01:42:23.662159 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:42:23.662125    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-hlnsl_97c56d5c-fa82-4ff9-b039-dd2839271bc7/istio-proxy/0.log"
Oct 31 01:42:24.071284 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:42:24.071194    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-mbzdk_738f0544-ef43-4c6c-a9ec-a20a362c482d/istio-proxy/0.log"
Oct 31 01:42:24.853742 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:42:24.853709    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-zrjvf_9b1dad6d-c028-4673-96f1-48c4fca807c4/istio-proxy/0.log"
Oct 31 01:42:25.234064 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:42:25.233990    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-64nxw_852f9567-a315-448d-b286-f90afbcaad43/payload-pre-processing/0.log"
Oct 31 01:42:26.148467 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:42:26.148432    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-mx9bj_cad8e534-8589-407a-bbcf-0bb145b80233/main/0.log"
Oct 31 01:42:26.743046 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:42:26.743013    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-s24sn_e2f2f0dd-a9a0-4335-9c33-683562fa8fbe/main/0.log"
Oct 31 01:42:26.940323 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:42:26.940291    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq7q_7c1e66f4-2552-43b1-96e0-272661898154/main/0.log"
Oct 31 01:42:27.137934 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:42:27.137907    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dddkj_b1988c46-0b7f-4edd-b441-ab408ad0a246/main/0.log"
Oct 31 01:42:27.335942 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:42:27.335915    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-bsnxt_c1398a6a-1ac9-48e1-a988-19c575e0cb91/main/0.log"
Oct 31 01:42:45.291082 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:42:45.291038    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:43:00.323170 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:43:00.323133    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9c8d8b4d5-z5sc9_0121470e-c1cd-459c-ba8b-0c6f8cac9c4f/manager/0.log"
Oct 31 01:43:00.635070 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:43:00.635038    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9c8d8b4d5-z5sc9_0121470e-c1cd-459c-ba8b-0c6f8cac9c4f/manager/0.log"
Oct 31 01:43:15.293225 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:43:15.293183    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:43:43.198946 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:43:43.198871    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9c8d8b4d5-z5sc9_0121470e-c1cd-459c-ba8b-0c6f8cac9c4f/manager/0.log"
Oct 31 01:43:43.544836 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:43:43.544754    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9c8d8b4d5-z5sc9_0121470e-c1cd-459c-ba8b-0c6f8cac9c4f/manager/0.log"
Oct 31 01:43:45.291661 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:43:45.291621    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:43:49.893407 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:43:49.893377    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pncsr_002d2da2-bb18-4667-98c8-1770b70d9985/global-pull-secret-syncer/0.log"
Oct 31 01:43:50.896390 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:43:50.896354    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g24ff_17401e80-fe41-417f-9840-acd7a68d17e3/konnectivity-agent/0.log"
Oct 31 01:43:51.131450 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:43:51.131418    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-253.ec2.internal_dd74f5dcc55b4e9100e01ca71d574990/haproxy/0.log"
Oct 31 01:43:59.733244 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:43:59.733213    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-k7vdf_e0686e3b-1c7a-42f1-b7be-7c85553be299/kuadrant-console-plugin/0.log"
Oct 31 01:44:00.199279 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.199250    4327 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9cckr/perf-node-gather-daemonset-4c2md"]
Oct 31 01:44:00.199616 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.199605    4327 cpu_manager.go:414] "containerMap: removing container" podUID="35b06a6a-0926-42de-8a8c-ba6aa58ef48a" containerName="payload-processing"
Oct 31 01:44:00.199663 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.199618    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35b06a6a-0926-42de-8a8c-ba6aa58ef48a" containerName="payload-processing"
Oct 31 01:44:00.199663 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.199628    4327 cpu_manager.go:414] "containerMap: removing container" podUID="8d65cd0c-49f1-4b1b-8872-6693bc6c982e" containerName="payload-pre-processing"
Oct 31 01:44:00.199663 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.199633    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d65cd0c-49f1-4b1b-8872-6693bc6c982e" containerName="payload-pre-processing"
Oct 31 01:44:00.199663 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.199643    4327 cpu_manager.go:414] "containerMap: removing container" podUID="6b74fa47-62f0-44e3-b56c-a52497d85960" containerName="istio-proxy"
Oct 31 01:44:00.199663 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.199648    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b74fa47-62f0-44e3-b56c-a52497d85960" containerName="istio-proxy"
Oct 31 01:44:00.199663 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.199656    4327 cpu_manager.go:414] "containerMap: removing container" podUID="6b74fa47-62f0-44e3-b56c-a52497d85960" containerName="istio-proxy"
Oct 31 01:44:00.199663 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.199661    4327 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b74fa47-62f0-44e3-b56c-a52497d85960" containerName="istio-proxy"
Oct 31 01:44:00.199921 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.199748    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="35b06a6a-0926-42de-8a8c-ba6aa58ef48a" containerName="payload-processing"
Oct 31 01:44:00.199921 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.199758    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d65cd0c-49f1-4b1b-8872-6693bc6c982e" containerName="payload-pre-processing"
Oct 31 01:44:00.199921 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.199764    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b74fa47-62f0-44e3-b56c-a52497d85960" containerName="istio-proxy"
Oct 31 01:44:00.199921 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.199770    4327 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b74fa47-62f0-44e3-b56c-a52497d85960" containerName="istio-proxy"
Oct 31 01:44:00.202752 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.202734    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-4c2md"
Oct 31 01:44:00.226963 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.226942    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9cckr\"/\"openshift-service-ca.crt\""
Oct 31 01:44:00.238437 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.238412    4327 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9cckr\"/\"default-dockercfg-l8p77\""
Oct 31 01:44:00.238546 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.238480    4327 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9cckr\"/\"kube-root-ca.crt\""
Oct 31 01:44:00.240174 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.240146    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9cckr/perf-node-gather-daemonset-4c2md"]
Oct 31 01:44:00.247069 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.247045    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn2gz\" (UniqueName: \"kubernetes.io/projected/e974b611-df7b-480a-8fca-d07430351a72-kube-api-access-wn2gz\") pod \"perf-node-gather-daemonset-4c2md\" (UID: \"e974b611-df7b-480a-8fca-d07430351a72\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-4c2md"
Oct 31 01:44:00.247154 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.247089    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e974b611-df7b-480a-8fca-d07430351a72-podres\") pod \"perf-node-gather-daemonset-4c2md\" (UID: \"e974b611-df7b-480a-8fca-d07430351a72\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-4c2md"
Oct 31 01:44:00.247213 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.247196    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e974b611-df7b-480a-8fca-d07430351a72-lib-modules\") pod \"perf-node-gather-daemonset-4c2md\" (UID: \"e974b611-df7b-480a-8fca-d07430351a72\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-4c2md"
Oct 31 01:44:00.247326 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.247307    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e974b611-df7b-480a-8fca-d07430351a72-sys\") pod \"perf-node-gather-daemonset-4c2md\" (UID: \"e974b611-df7b-480a-8fca-d07430351a72\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-4c2md"
Oct 31 01:44:00.247383 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.247359    4327 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e974b611-df7b-480a-8fca-d07430351a72-proc\") pod \"perf-node-gather-daemonset-4c2md\" (UID: \"e974b611-df7b-480a-8fca-d07430351a72\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-4c2md"
Oct 31 01:44:00.348794 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.348759    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e974b611-df7b-480a-8fca-d07430351a72-lib-modules\") pod \"perf-node-gather-daemonset-4c2md\" (UID: \"e974b611-df7b-480a-8fca-d07430351a72\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-4c2md"
Oct 31 01:44:00.348962 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.348813    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e974b611-df7b-480a-8fca-d07430351a72-sys\") pod \"perf-node-gather-daemonset-4c2md\" (UID: \"e974b611-df7b-480a-8fca-d07430351a72\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-4c2md"
Oct 31 01:44:00.348962 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.348839    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e974b611-df7b-480a-8fca-d07430351a72-proc\") pod \"perf-node-gather-daemonset-4c2md\" (UID: \"e974b611-df7b-480a-8fca-d07430351a72\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-4c2md"
Oct 31 01:44:00.348962 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.348883    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn2gz\" (UniqueName: \"kubernetes.io/projected/e974b611-df7b-480a-8fca-d07430351a72-kube-api-access-wn2gz\") pod \"perf-node-gather-daemonset-4c2md\" (UID: \"e974b611-df7b-480a-8fca-d07430351a72\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-4c2md"
Oct 31 01:44:00.348962 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.348908    4327 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e974b611-df7b-480a-8fca-d07430351a72-podres\") pod \"perf-node-gather-daemonset-4c2md\" (UID: \"e974b611-df7b-480a-8fca-d07430351a72\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-4c2md"
Oct 31 01:44:00.348962 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.348934    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e974b611-df7b-480a-8fca-d07430351a72-lib-modules\") pod \"perf-node-gather-daemonset-4c2md\" (UID: \"e974b611-df7b-480a-8fca-d07430351a72\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-4c2md"
Oct 31 01:44:00.348962 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.348957    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e974b611-df7b-480a-8fca-d07430351a72-proc\") pod \"perf-node-gather-daemonset-4c2md\" (UID: \"e974b611-df7b-480a-8fca-d07430351a72\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-4c2md"
Oct 31 01:44:00.349274 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.348944    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e974b611-df7b-480a-8fca-d07430351a72-sys\") pod \"perf-node-gather-daemonset-4c2md\" (UID: \"e974b611-df7b-480a-8fca-d07430351a72\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-4c2md"
Oct 31 01:44:00.349274 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.349018    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e974b611-df7b-480a-8fca-d07430351a72-podres\") pod \"perf-node-gather-daemonset-4c2md\" (UID: \"e974b611-df7b-480a-8fca-d07430351a72\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-4c2md"
Oct 31 01:44:00.372507 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.372481    4327 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn2gz\" (UniqueName: \"kubernetes.io/projected/e974b611-df7b-480a-8fca-d07430351a72-kube-api-access-wn2gz\") pod \"perf-node-gather-daemonset-4c2md\" (UID: \"e974b611-df7b-480a-8fca-d07430351a72\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-4c2md"
Oct 31 01:44:00.514512 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.514433    4327 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-4c2md"
Oct 31 01:44:00.822131 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:00.822101    4327 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9cckr/perf-node-gather-daemonset-4c2md"]
Oct 31 01:44:00.822653 ip-10-0-128-253 kubenswrapper[4327]: W1031 01:44:00.822626    4327 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode974b611_df7b_480a_8fca_d07430351a72.slice/crio-2b23a45106d5cbbd98dfa7b1038048d9f8d53ee2ed980a14ccb2d6a0f3474d7b WatchSource:0}: Error finding container 2b23a45106d5cbbd98dfa7b1038048d9f8d53ee2ed980a14ccb2d6a0f3474d7b: Status 404 returned error can't find the container with id 2b23a45106d5cbbd98dfa7b1038048d9f8d53ee2ed980a14ccb2d6a0f3474d7b
Oct 31 01:44:01.205064 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:01.205026    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-4c2md" event={"ID":"e974b611-df7b-480a-8fca-d07430351a72","Type":"ContainerStarted","Data":"5775ddb42bc177f3aa6d86864d4250c57266cf1d0600d7cefa9b9da255b82218"}
Oct 31 01:44:01.205253 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:01.205084    4327 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-4c2md" event={"ID":"e974b611-df7b-480a-8fca-d07430351a72","Type":"ContainerStarted","Data":"2b23a45106d5cbbd98dfa7b1038048d9f8d53ee2ed980a14ccb2d6a0f3474d7b"}
Oct 31 01:44:01.205253 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:01.205133    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-4c2md"
Oct 31 01:44:01.247539 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:01.247485    4327 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-4c2md" podStartSLOduration=1.247472247 podStartE2EDuration="1.247472247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:01.245844038 +0000 UTC m=+5101.467879580" watchObservedRunningTime="2026-10-31 01:44:01.247472247 +0000 UTC m=+5101.469507805"
Oct 31 01:44:03.494169 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:03.494133    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-czprd_4a336e35-d2f5-4fac-9ac2-6784b9b34108/node-exporter/0.log"
Oct 31 01:44:03.606778 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:03.606745    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-czprd_4a336e35-d2f5-4fac-9ac2-6784b9b34108/kube-rbac-proxy/0.log"
Oct 31 01:44:03.743854 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:03.743817    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-czprd_4a336e35-d2f5-4fac-9ac2-6784b9b34108/init-textfile/0.log"
Oct 31 01:44:05.561147 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:05.561116    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-srmtb_6e3c9fbf-af3d-49e2-8a7e-0848b39fc181/prometheus-operator/0.log"
Oct 31 01:44:05.671028 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:05.670996    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-srmtb_6e3c9fbf-af3d-49e2-8a7e-0848b39fc181/kube-rbac-proxy/0.log"
Oct 31 01:44:05.791131 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:05.791101    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-j9k2n_dc8f439a-7709-48f9-948a-eb6b76c117e1/prometheus-operator-admission-webhook/0.log"
Oct 31 01:44:05.914863 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:05.914825    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-f758cf87d-dh7dw_b0c912b9-caee-4949-9010-5cbe2a5664c5/telemeter-client/0.log"
Oct 31 01:44:06.030898 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:06.030866    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-f758cf87d-dh7dw_b0c912b9-caee-4949-9010-5cbe2a5664c5/reload/0.log"
Oct 31 01:44:06.145476 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:06.145450    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-f758cf87d-dh7dw_b0c912b9-caee-4949-9010-5cbe2a5664c5/kube-rbac-proxy/0.log"
Oct 31 01:44:07.220013 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:07.219988    4327 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-4c2md"
Oct 31 01:44:10.618825 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:10.618799    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-4k28w_93aea69b-c34d-4a21-acf8-a71d62fe8ef2/download-server/0.log"
Oct 31 01:44:14.843053 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:14.843022    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v8ft8_79ab34ed-5163-4cff-8cf4-170c3445e655/dns/0.log"
Oct 31 01:44:14.963735 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:14.963701    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v8ft8_79ab34ed-5163-4cff-8cf4-170c3445e655/kube-rbac-proxy/0.log"
Oct 31 01:44:15.082492 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:15.082460    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9d69r_ec6a6238-81be-4a82-b346-b4c9fe3aefa4/dns-node-resolver/0.log"
Oct 31 01:44:15.301573 ip-10-0-128-253 kubenswrapper[4327]: E1031 01:44:15.301530    4327 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vv2qm" containerName="extractor"
Oct 31 01:44:16.519701 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:16.519646    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-564986b4bd-526b7_e1b3a4a9-bed0-4574-9b0b-35c1ea45a49f/registry/0.log"
Oct 31 01:44:16.638515 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:16.638473    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-htbtd_95f3be30-2d65-4373-9623-155c841dca11/node-ca/0.log"
Oct 31 01:44:18.725360 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:18.725325    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-frj8v_97aa8115-64f5-478f-93b1-276dc60acd6a/discovery/0.log"
Oct 31 01:44:18.759035 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:18.759007    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-94bcdf98f-g9zcg_3abd5af3-5337-4948-849b-b6d2976f269e/kube-auth-proxy/0.log"
Oct 31 01:44:19.103667 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:19.103636    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-hlnsl_97c56d5c-fa82-4ff9-b039-dd2839271bc7/istio-proxy/0.log"
Oct 31 01:44:19.257706 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:19.257658    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-mbzdk_738f0544-ef43-4c6c-a9ec-a20a362c482d/istio-proxy/0.log"
Oct 31 01:44:19.532898 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:19.532804    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-zrjvf_9b1dad6d-c028-4673-96f1-48c4fca807c4/istio-proxy/0.log"
Oct 31 01:44:19.663961 ip-10-0-128-253 kubenswrapper[4327]: I1031 01:44:19.663902    4327 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-64nxw_852f9567-a315-448d-b286-f90afbcaad43/payload-pre-processing/0.log"